Analysis of Online Reviews for Evaluating the Quality of Cultural Tourism

Online reviews have long represented a valuable source for data analysis in the tourism field, but these data sources have been mostly studied in terms of the numerical ratings offered by the review platforms.

In a recent article (available as full open-access) and a respective blog post, we explored if social media and online review platforms can be a good source of quantitative evaluation of service quality of cultural venues, such as museums, theaters and so on. Our paper applies automatic analysis of online reviews, by comparing two different automated analysis approaches to evaluate which of the two is more adequate for assessing the quality dimensions. The analysis covers user-generated reviews over the top 100 Italian museums. 

Specifically, we compare two approaches:

  • a ‘top-down’ approach that is based on a supervised classification based upon strategic choices defined by policy makers’ guidelines at the national level; 
  • a ‘bottom-up’ approach that is based on an unsupervised topic model of the online words of reviewers.

The misalignment of the results of the ‘top-down’ strategic studies and ‘bottom-up’ data-driven approaches highlights how data science can offer an important contribution to decision making in cultural tourism.  Both the analysis approaches have been applied to the same dataset of 14,250 Italian reviews.

We identified five quality dimensions that follow the ‘top-down’ perspective: Ticketing and Welcoming, Space, Comfort, Activities, and Communication. Each of these dimensions has been considered as a class in a classification problem over user reviews. The top down approach allowed us to tag each review as descriptive of one of those 5 dimensions. Classification has been implemented both as a machine learning classification problem (using BERT, accuracy 88%) and as and keyword-based tagging (accuracy 80%).

The ‘bottom-up’ approach has been implemented through an unsupervised topic modelling approach, namely LDA (Latent Dirichlet Allocation), implemented and tuned over a range up to 30 topics. The best ‘bottom-up’ model we selected identifies 13 latent dimensions in review texts. We further integrated them in 3 main topics: Museum Cultural Heritage, Personal Experience and Museum Services.

The ‘top-down’ approach (based on a set of keywords defined from the standards issued by the policy maker) resulted in 63% of online reviews that did not fit into any of the predefined quality dimension.

63% of the reviews could not be assessed against the official top-down service quality categories.

The ‘bottom-up’ data-driven approach overcomes this limitation by searching for the aspects of interest using reviewers’ own words. Indeed, usually museum reviews discuss more about a museum’s cultural heritage aspects (46% average probability) and personal experiences (31% average probability) than the services offered by the museum (23% average probability).

Among the various quantitative findings of the study, I think the most important point is that the aspects considered as quality dimensions by the decision maker can be highly different from those aspects perceived as quality dimensions by museum visitors.

You can find out more about this analysis by reading the full article published online as open-access, or this longer blog post . The full reference to the paper is:

Agostino, D.; Brambilla, M.; Pavanetto, S.; Riva, P. The Contribution of Online Reviews for Quality Evaluation of Cultural Tourism Offers: The Experience of Italian Museums. Sustainability 2021, 13, 13340.

Bran Selic, Mark Van Den Brand and Marco Brambilla: Panel on State of Modeling.

Within the Modelsward conference held at ESEO in Angers, France, the warm up session has been delegated to a panel on the state of modelling, where panelists Bran Selic, Mark Van Den Brand and myself discussed about their vision on modelling.

Plenty of good points came up both from the panelists and the audience.

The main message from Bran Selic was:

  • Non-functional modelling is wrong, both in terminology and semantic senses
  • the so called “ities” are more than 50, and still each of them needs to be covered with different techniques and tools
  • The term implies a second-order importance, and a negative definition which are bad by themselves
  • Furthermore, you cannot cover them separately wrt functional requirements, because it’s not true you can cover them through aspect-oriented or separation of concern. For instance, you cannot put in “reliability” to a system after you have covered the functional aspects only
  • So, best to call them “qualities” of the system. Even in ancient Greece, quality was undistinguishable from the thing itself

On the other side, Mark Van Den Brand was pointing to:

  •  the risk for software engineers to become obsolete, because every domain expert is going to build his own tools and languages by himself
  • the software engineers then should become more interdisciplinary, while not delve into becoming domain experts of some sort.
My provocative message at the panel was that:
  • Modeling is dead! In the sense we cannot expect
  • We should move to Un-Modeling Practices, that is: remove modelling as a tool for everybody, using modelling tools as experts, and let people enjoy only the little bit they deserve, without forcing MDD frameworks, which are usually not easily accepted
  • Modeling should be used under the hood and each actor should be shown the proper tool (including programmers, who are not easily buying the model-driven approaches at all)
  • This will not be possible until language and tool designers will be software engineers only (just think at the terrible modelling tools we are able to build).
  • Here is the slides I used for my pitch:
I think all this poses extremely complex challenges to the modelling and software engineering community. It’s up to us to keep up with these challenges, or become obsolete (aka. remain a very small niche in the software development world).

To keep updated on my activities you can subscribe to the RSS feed of my blog or follow my twitter account (@MarcoBrambi).