In search of new product ideas: Identifying ideas in online communities by machine learning and text mining

TY - JOUR

T1 - In search of new product ideas

T2 - Identifying ideas in online communities by machine learning and text mining

AU - Christensen, Kasper

AU - Nørskov, Sladjana

AU - Frederiksen, Lars

AU - Scholderer, Joachim

PY - 2017

Y1 - 2017

N2 - Online communities are attractive sources of ideas relevant for new product development and innovation. However, making sense of the ‘big data’ in these communities is a complex analytical task. A systematic way of dealing with these data is needed to exploit their potential for boosting companies' innovation performance. We propose a method for analysing online community data with a special focus on identifying ideas. We employ a research design where two human raters classified 3,000 texts extracted from an online community, according to whether the text contained an idea. Among the 3,000, 137 idea texts and 2,666 non-idea texts were identified. The human raters could not agree on the remaining 197 texts. These texts were omitted from the analysis. The remaining 2,803 texts were processed by using text mining techniques and used to train a classification model. We describe how to tune the model and which text mining steps to perform. We conclude that machine learning and text mining can be useful for detecting ideas in online communities. The method can help researchers and firms identify ideas hidden in large amounts of texts. Also, it is interesting in its own right that machine learning can be used to detect ideas.

AB - Online communities are attractive sources of ideas relevant for new product development and innovation. However, making sense of the ‘big data’ in these communities is a complex analytical task. A systematic way of dealing with these data is needed to exploit their potential for boosting companies' innovation performance. We propose a method for analysing online community data with a special focus on identifying ideas. We employ a research design where two human raters classified 3,000 texts extracted from an online community, according to whether the text contained an idea. Among the 3,000, 137 idea texts and 2,666 non-idea texts were identified. The human raters could not agree on the remaining 197 texts. These texts were omitted from the analysis. The remaining 2,803 texts were processed by using text mining techniques and used to train a classification model. We describe how to tune the model and which text mining steps to perform. We conclude that machine learning and text mining can be useful for detecting ideas in online communities. The method can help researchers and firms identify ideas hidden in large amounts of texts. Also, it is interesting in its own right that machine learning can be used to detect ideas.

UR - http://www.scopus.com/inward/record.url?scp=85006761522&partnerID=8YFLogxK

U2 - 10.1111/caim.12202

DO - 10.1111/caim.12202

M3 - Journal article

SN - 0963-1690

VL - 26

SP - 17

EP - 30

JO - Creativity and Innovation Management

JF - Creativity and Innovation Management

IS - 1

ER -