Bots in the Election

4:15:00 PM
by Unknown

In the Kno.e.sis Center at Wright State University, we continue to refine our Twitris technology (Licensed by Cognovi Labs LLC) for collective social intelligence to analyze social media (esp. Twitter) in real time. Kno.e.sis and Cognovi Labs teamed up with the Applied Policy Research Institute (APRI) early in the year and created some tools to monitor the debates. See press coverage on TechCrunch. From the time that we first began following the nominees on Twitter, one thing became clear: Donald Trump was considerably more popular than his competition during the primaries as well as the general election. To be honest, I had never considered the possibility that social bots may have been playing a role in this popularity.


After the conclusion of the first debate all parties who had watched our "Debate Dashboard" were shocked not just by the volume of tweets but by the sentiment and emotion that appeared more positive for Trump than Clinton. When we investigated news from major media outlets, we became more and more concerned that our tool had some serious flaws. Due to articles we had seen discussing the large support Trump had on Twitter, we decided to focus on sentiment. Up until a few days before the election, we continued to update and improve our sentiment analysis algorithm.   


Notwithstanding the improvements in precision to our sentiment classifier, we continued to see Trump as the clear leader. As the debates came and went, our data remained consistent. We began an urgent quest for reason. We added gender analysis because media outlets were telling us that women were down on Trump and would be a major force in the election. Our analysis did not show this, despite having 96% precision in determining female and male users. We developed a proprietary process to separate users into left-leaning and right-leaning users. We could even say whether a user was strongly or loosely associated with a particular political party. Unfortunately, analyzing the data based on political association didn't help either. Surprisingly many strongly left leaning users were anti-Hillary, just a bit behind the right-leaning users.


After the second debate, we began to see many articles pop up about social bots. Once we began to look more into the issue, we found many articles from early in the year talking about Trump's "Bot Army" (Trumps' Biggest Lie? The Size of His Twitter Following). We had our aha moment. In that article, there is a reference to The Atlantic's use of a tool called BotOrNot. We decided to attempt to use this or some similar tool during the last debate to remove users with bot accounts and analyze the remaining data.


BotOrNot is a tool developed at the University of Indiana, Bloomington with collaboration from the University of Southern California, Marina Del Rey. Their tool creates over one thousand metrics by looking at the user account and analyzing retweets, hashtags, metadata, etc. The tool performed extremely well in the DARPA Twitter Bot Challenge, having correctly identifying all of the known bots (though it did incorrectly mark some additional users as bots). We were excited to learn that they had made their tool available through an API endpoint and decided to run our tweets through the system to test the speed at which we could process users. Twitris at this point was processing nearly 35 tweets per second for the election analysis alone, and it was clear very quickly that their service would not be able to handle the volume of data we would be consuming.


Though the final presidential debate was only several days away, we still had hope that we would find an answer. We saw Prof. Philip Howard, from the University of Oxford, mention in The Washington Post that they considered any user that tweets more than 50 times in one day a bot. It would be relatively simple to create an index of users and simply increment the count per tweet and then check the index quickly as the tweets roll through. We may have done this if our team had been free at the time. Some were working on bug fixes, others on improving sentiment, and still others were working to fix some infrastructure issues that we were experiencing at the time. Our corpus of tweets for the election campaign was on its way to exceeding 60 million tweets. A robust implementation would have required more time than anyone had to offer.


I suppose that now is a better time than never to define what a "bot" is. Phil Howard says, in that Washington Post article mentioned above, that "A Twitter bot is nothing more than a program that automatically posts messages to Twitter and/or auto-retweets the messages of others". I personally like this definition, but it leaves a little wiggle room where tools like BotOrNot, which focus primarily on the user accounts, are considered. A Twitter user can be a real, living, breathing, human being and still exhibit bot-like tweeting habits. I think that this is the reason that Howard's group settled on the 50-tweets-per-day stat instead of relying on a classifier. A user can tweet for themselves part of the time, but still have some automation process that tweets certain things on their behalf. There are many reasons that someone would do this, to increase their Klout score (imagine a LuLaRoe seller, YouTuber, or Blogger), for example. There are many companies that you can pay to automate this kind of activity for you. Some, like Linkis' "Convey" (more on this later) work by finding influential tweets and tweeting them on your behalf. These tweets are fairly easy to spot because they attach "via @c0nvey" to the end of the original tweet.


In the end, what we did was to develop a system that is able to quickly and accurately weed out tweets that were not authored by humans, even if the user account is owned by an actual human. Let's see how well our system stacks up to the BotOrNot service. We collected all of the bots found over one fifteen minute period and ran them through BotOrNot. 67.16% of the accounts from the tweets we labeled were determined by BotOrNot to be “bot” owned accounts. I was a little disappointed by this, so I took a look at the users that BotOrNot dismissed as human. In the first pass, I decided to apply Howard's 50-tweets-per-day rule. Doing this increased the percentage of accurately labeled bot tweets to 73.88%. One of the screen names found among this group actually contained the word "bot" in it. The most important thing for us at this point was to make sure that we weren’t getting a lot of false positives. We looked at each account one by one; we looked at each tweet in our system, both those classified as "likely bot" and "likely human".


 
Figure: Real-time labeling of bot and human tweets in Twitris




Many of the users with a low tweet-per-day usage had a combination of the two, and looking at the tweets that were labeled "likely bot", nearly all of them contained "via @c0nvey". Digging a bit further, I found that the Convey service has been accused of tweeting on behalf of unwitting users in the past. Here's a thread from one Twitter user talking about his experience:
   c0nvey.jpg


So, our system is able to accurately detect automated (bot) tweets despite the fact that the user is not a bot. Our average detection rate also looks reasonable compared to other services. We experience bot traffic at a rate of about a 5% per day in our election campaign, though there are days that get to nearly 8%.


Moving Forward (oh, and Fake News)


Elections come and go. Once they are gone, we seek to apply our findings to future work. Being able to detect bots is great; but, what else can we do with that? Well, post-election we have learned the term “fake news” (with content that was entirely made up, not grounded in truth or reality), something that not too many people were concerned about before: see the Google Trend.
fakenews.jpg
Where does all of this fake news come from? There have been troves of news reports blaming Facebook and Twitter for altering the outcome of the election. Obviously, Facebook and Twitter themselves weren’t creating pro-Trump news (Facebook in particular was accused of killing pro-Trump trends); however, some blame these companies for not alerting people that the news that they were “allowing” to spread was fake. I don’t think that that is fair. These companies are reliant on the fact that they don’t publish news (for more, read this article on Facebook, the “News Feed”, and the Communications Decency Act 230) to avoid lawsuits.


During the evaluation of our bot detection system, we noticed something interesting. A large majority of the bot-labeled tweets contained links to dubious looking news stories. Because of our ability to identify these bot tweets, we can exclude them from analysis when considering a brand-centered campaign (like Samsung during the Note 7 battery “situation”). I think that it is important to note that, despite the elimination of “fake news” and “bot tweets” from our analysis, Trump was always winning. We saw the same thing with the Brexit referendum earlier this year, where Twitris helped us to correctly predict the Brexit outcome before the polls closed. There is clear evidence to support the fact that high tweet volume translates to success (except Bernie Sanders, but there may be, uh-hum, other reasons for that). It seems to me that for bots and “fake news” to have swayed the election, they would have needed to be ready to go as soon as Trump announced that he was running, but what we have seen is that he was always ahead.


We will continue to find new ways to leverage everything that we learned from the 2016 Election. If you want to stay up to date on our analysis, please sign up to receive CognoviLabs’ newsletter at www.CognoviLabs.com or join Kno.e.sis on FB, and while you are there check the other post-election analysis.

You Might Also Like

24 comments

  1. M88 Casino Review 2021 | Up to C$100 Welcome Bonus - thakasino
    M88 casino welcome bonus | 100% first deposit bonus up to C$100 | 100% first deposit bonus up to C$100 | m88 ทางเข้า 100% 우리카지노 first 188bet deposit bonus up to C$100 | 100% first deposit bonus up

    ReplyDelete
  2. I really enjoy your web’s topic. Very creative and friendly for users. Definitely bookmark this and follow it everyday. 바카라사이트

    ReplyDelete
  3. Hello, nice to read your all blogs and learned a ot of new knowledge. Thank you! 토토사이트

    ReplyDelete
  4. Your blog has a lot of useful information. I am very lucky to have found your blog. We look forward to seeing many more diverse opinions in the future. 바카라사이트닷컴

    ReplyDelete
  5. You’re doing a great job, keep it up doing a great work like this one. 파워볼사이트닷컴

    ReplyDelete
  6. Very efficiently written information. It will be valuable to everyone who utilizes it, as well as yours truly. Keep doing what you are doing – for sure i will check out more posts Feel free to visit my website; 바카라사이트

    ReplyDelete
  7. I’m very pleased to discover this site. I want to to thank you for ones time for this particularly wonderful read!! I definitely savored every part of it and i also have you saved as a favorite to see new information on your blog. 메이저토토사이트

    ReplyDelete
  8. Nice article I agree with this.Your blog really nice. Its sound really good
    토토사이트웹

    ReplyDelete
  9. Your article is very interesting. I think this article has a lot of information needed, looking forward to your new posts.
    바카라사이트윈

    ReplyDelete
  10. I really appreciate all the hard work you put into this great site. Thankyou so much.. 카지노사이트

    ReplyDelete
  11. Nice post. It’s always interesting to read articles here from other authors and practice something from their web sites 온라인카지노

    ReplyDelete
  12. This is a great inspiring article. I am pretty much pleased with your good work. Keep it up, Thanks.. 카지노

    ReplyDelete
  13. I am always searching online for articles that can help me. I think you made some good points. Keep working, great job ! 바카라

    ReplyDelete
  14. I'm happy to see your website for valuable information.Why didn't this accident happen earlier? I was surprised!For me, it's bookmarked.

    온라인카지노
    메이저토토사이트
    토토사이트

    ReplyDelete
  15. I just want to say that your article is amazing. The clarity of your pretense is truly wonderful, and I can guess you are an expert on this subject. If you don't mind, I'll get an RSS feed and update you on future posts. Thank you very much.Please continue to enjoy your work.

    스포츠토토
    토토
    바카라사이트

    ReplyDelete
  16. Can I simply say what a relief to find somebody who really understands what they are discussing on the web.
    스포츠토토핫

    ReplyDelete
  17. Aw, this was an extremely nice post. Taking a few minutes and actual effort to generate a top
    notch article.
    메이저토토사이트

    ReplyDelete
  18. Thank you for sharing this information. I read your blog and I can't not stopped my self to read your full blog. Again Thanks and Best of luck your next Blog in future.
    토토사이트웹

    ReplyDelete
  19. Excellent information on your blog, thank you for taking the time to share with us. 슬롯머신

    ReplyDelete
  20. I blog often ɑnd I sеriously apⲣreciate yօur c᧐ntent.
    Yߋur article һaѕ truly peaked my іnterest. I am ցoing to book mark
    your site and keep checking ffor neѡ details аbout once a wеek.
    I subscribed to ʏоur RSS feed tοo.

    My webpage :: https://www.casinoholic.info 바카라사이트

    ReplyDelete

  21. Cool thing to have read this article !
    검증카지노

    click here now booter
    온라인카지노

    ReplyDelete
  22. Everything is very open with a very clear explanation. sportstoto It was definitely informative. Your site is very useful.
    Thank you for sharing!

    스포츠토토
    파워볼게임
    안전놀이터

    ReplyDelete