skip to main content
10.1145/2998181.2998197acmconferencesArticle/Chapter ViewAbstractPublication PagescscwConference Proceedingsconference-collections
research-article
Open Access

Crowdsourcing as a Tool for Research: Implications of Uncertainty

Published:25 February 2017Publication History

ABSTRACT

Numerous crowdsourcing platforms are now available to support research as well as commercial goals. However, crowdsourcing is not yet widely adopted by researchers for generating, processing or analyzing research data. This study develops a deeper understanding of the circumstances under which crowdsourcing is a useful, feasible or desirable tool for research, as well as the factors that may influence researchers' decisions around adopting crowdsourcing technology. We conducted semi-structured interviews with 18 researchers in diverse disciplines, spanning the humanities and sciences, to illuminate how research norms and practitioners' dispositions were related to uncertainties around research processes, data, knowledge, delegation and quality. The paper concludes with a discussion of the design implications for future crowdsourcing systems to support research.

References

  1. Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating online labor markets for experimental research: Amazon.com's mechanical turk. Political Analysis 20, 3 (2012), 351--368.Google ScholarGoogle ScholarCross RefCross Ref
  2. Jeremy P. Birnholtz and Matthew J. Bietz. 2003. Data at Work: Supporting Sharing in Science and Engineering. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work (GROUP '03). ACM, New York, NY, USA, 339--348. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. David Blumenthal, Eric G Campbell, Manjusha Gokhale, Recai Yucel, Brian Clarridge, Stephen Hilgartner, and Neil a Holtzman. 2006. Data withholding in genetics and the other life sciences: prevalences and predictors. Academic Medicine 81, 2 (2006), 137--145.Google ScholarGoogle ScholarCross RefCross Ref
  4. Sara Bly. 1998. Special Section on Collaboratories. interactions 5, 3 (May 1998), 31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. John Bohannon. 2016. Psychologists grow increasingly dependent on online research subjects. (2016). Retrieved June 7, 2016 from http://www.sciencemag.org/news/2016/06/psychologistsgrow-increasingly-dependent-online-research-subjects.Google ScholarGoogle Scholar
  6. Rick Bonney, Caren B. Cooper, Janis Dickinson, Steve Kelling, Tina Phillips, Kenneth V. Rosenberg, and Jennifer Shirk. 2009. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience 59, 11 (2009), 977--984.Google ScholarGoogle ScholarCross RefCross Ref
  7. Anne Bowser, Derek Hansen, Jennifer Preece, Yurong He, Carol Boston, and Jen Hammock. 2014. Gamifying Citizen Science: A Study of Two User Groups. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work (CSCW Companion '14). ACM, New York, NY, USA, 137--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Anne Bowser-Livermore and Andrea Wiggins. 2015. Privacy in Participatory Research: Advancing Policy to support Human Computation. Human Computation 2, 1 (2015), 19--44.Google ScholarGoogle ScholarCross RefCross Ref
  9. Alex Bowyer, Chris Lintott, Greg Hines, Campbell Allan, and Ed Paget. 2015. Panoptes, a Project Building Tool for Citizen Science. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP '15). AAAI, San Diego, CA, USA, 1--2.Google ScholarGoogle Scholar
  10. H.K. Burgess, L.B. DeBey, H.E. Froehlich, N. Schmidt, E.J. Theobald, A.K. Ettinger, J. HilleRisLambers, J. Tewksbury, and J.K. Parrish. 2016. The science of citizen science: Exploring barriers to use as a primary research tool. Biological Conservation (2016), 1--8.Google ScholarGoogle Scholar
  11. Carolin Cardamone, Kevin Schawinski, Marc Sarzi, Steven P. Bamford, Nicola Bennert, C. M. Urry, Chris Lintott, William C. Keel, John Parejko, Robert C. Nichol, Daniel Thomas, Dan Andreescu, Phil Murray, M. Jordan Raddick, Anze Slosar, Alex Szalay, and Jan Vandenberg. 2009. Galaxy Zoo Green Peas: Discovery of a class of compact extremely star-forming galaxies. Monthly Notices of the Royal Astronomical Society 399, 3 (2009), 1191--1205.Google ScholarGoogle ScholarCross RefCross Ref
  12. Casey Trees. 2016. Citizen Science. (2016). Retrieved January 15, 2016 from http://caseytrees.org/get-involved/citizen-science/.Google ScholarGoogle Scholar
  13. Karin Knorr Cetina. 1999. Epistemic Communities: How the Sciences Make Knowledge. Harvard Education Press, Cambridge, MA.Google ScholarGoogle Scholar
  14. Sayan Chakraborti, Naveen Yadav, Carolin Cardamone, and Alak Ray. 2012. Radio Detection of Green Peas: Implications for Magnetic Fields in Young Galaxies. The Astrophysical Journal 746, 1 (2012), L6.Google ScholarGoogle ScholarCross RefCross Ref
  15. Jeffrey P Cohn. 2008. Citizen Science : Can Volunteers Do Real Research? BioScience 58, 3 (2008), 192--197.Google ScholarGoogle ScholarCross RefCross Ref
  16. Harry Collins. 1992. Changing Order: Replication and Induction in Scientific Practice. University of Chicago Press, Chicago, IL.Google ScholarGoogle Scholar
  17. Harry Collins and Robert Evans. 2007. Rethinking Expertise. University of Chicago Press.Google ScholarGoogle Scholar
  18. Colorado State University. 2016. Citsci.Org. (2016). Retrieved January 15, 2016 from http://www.citsci.org.Google ScholarGoogle Scholar
  19. Caren Cooper. 2016. Zen in the Art of Citizen Science: Apps for Collective Discovery and the 1 Percent Rule of the Web. (2016). Retrieved January 15, 2016 from http://blogs.scientificamerican.com/guest-blog/zen-inthe-art-of-citizen-science-apps-for-collectivediscovery-and-the-1-percent-rule-of-the-web/.Google ScholarGoogle Scholar
  20. Juliet M. Corbin and Anselm Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3rd ed.). Sage, Thousand Oaks, CA.Google ScholarGoogle Scholar
  21. M. L. Cornwell and L. M. Campbell. 2012. Co-producing conservation and knowledge: Citizen-based sea turtle monitoring in North Carolina, USA. Social Studies of Science 42, 1 (2012), 101--120.Google ScholarGoogle ScholarCross RefCross Ref
  22. Joe Cox, Eun Young Oh, Brooke Simmons, Chris Lintott, Karen Masters, Anita Greenhill, Gary Graham, and Kate Holmes. 2015. Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects. IEEE Computing in Science & Engineering 17, 4 (2015), 28--41.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. M. H. Cragin, C. L. Palmer, J. R. Carlson, and M. Witt. 2010. Data sharing, small science and institutional repositories. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 368, 1926 (2010), 4023--4038.Google ScholarGoogle ScholarCross RefCross Ref
  24. Schlagwein Daniel and Daneshgar Farhad. 2014. User requirements of a crowdsourcing platform for researchers : findings from a series of focus groups. In Proceedings of the 2014 Pacific Asia Conference on Information Systems. 1--11.Google ScholarGoogle Scholar
  25. Janet Dickinson and Rick Bonney. 2012. Citizen Science : Public Participation in Environmental Research. Cornstock Pub. Associates, Ithaca, NY.Google ScholarGoogle Scholar
  26. Michael Dummett. 1991. The Logical Basis of Metaphysics. Duckworth, London, UK.Google ScholarGoogle Scholar
  27. Kevin Dunbar. 2000. How Scientists Think in the Real World. Journal of Applied Developmental Psychology 21, 1 (2000), 49--58.Google ScholarGoogle ScholarCross RefCross Ref
  28. Alexandra Eveleigh, Charlene Jennett, Ann Blandford, Philip Brohan, and Anna L. Cox. 2014. Designing for Dabblers and Deterring Drop-outs in Citizen Science. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2985--2994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Elizabeth J Farnsworth, Miyoko Chu, W John Kress, Amanda K Neill, Jason H Best, John Pickering, Robert D Stevenson, Gregory W Courtney, John K V Dyk, and Aaron M Ellison. 2013. Next-Generation Field Guides. BioScience 63, 11 (2013), 891--899.Google ScholarGoogle ScholarCross RefCross Ref
  30. Karen Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics 37, 2 (June 2011), 413--420. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Amy Freitag and Max Pfeffer. 2013. Process, Not Product: Investigating Recommendations for Improving Citizen Science Success. PLoS One 8, 5 (2013), e64079.Google ScholarGoogle ScholarCross RefCross Ref
  32. Bruno S. Frey and Reto Jegen. 2001. Motivation Crowding Theory. Journal of Economic Surveys 15, 5 (2001), 589--611.Google ScholarGoogle ScholarCross RefCross Ref
  33. Jay R Galbraith. 1974. Organization Design: An Information Processing View. Interfaces 4, 3 (1974), 28--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Thomas F. Giergyn. 1999. Cultural Boundaries of Science: Credibility on the Line. University of Chicago Press, Chicago, IL.Google ScholarGoogle Scholar
  35. Thomas F Gieryn. 1983. Boundary-Work and the Demarcation of Science From Non-Science: Strains and Interests in Professional Ideologies of Scientists. American Sociological Review 48, 6 (1983), 781--795.Google ScholarGoogle ScholarCross RefCross Ref
  36. Elizabeth Goodman and Janet Vertesi. 2012. Design for X?: Distribution Choices and Ethical Design. In Extended Abstracts of the 30th Annual ACM Conference on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 81--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. David Grier. 2007. When Computers Were Humans. Princeton University Press, Princeton, NJ. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Katie Hafner and Matthew Lyon. 1996. Where Wizards Stay up Late: the Origins of the Internet. Simon & Schuster, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Muki Haklay. 2013. Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice. Springer, Berlin, Chapter Citizen Science and Volunteer Geographic Information: Overview and Typology of Participation, 105--122.Google ScholarGoogle Scholar
  40. Harvard School of Engineering and Applied Sciences. 2016. Privacy Tools for Sharing Research Data. (2016). Retrieved January 15, 2016 from http://privacytools.seas.harvard.edu.Google ScholarGoogle Scholar
  41. Imperial College London. 2016. EpiCollect.net Website. (2016). Retrieved January 15, 2016 from http://www.epicollect.net.Google ScholarGoogle Scholar
  42. Yuri I. Izotov, Natalia G. Guseva, and Trinh X. Thuan. 2011. Green Pea Galaxies and Cohorts: Luminous Compact Emission-Line Galaxies in the Sloan Digital Sky Survey. The Astrophysical Journal 728, 2 (2011), 161.Google ScholarGoogle ScholarCross RefCross Ref
  43. Corey Jackson, Gabriel Mugar, Kevin Crowston, and Carsten Osterlund. 2016. Encouraging Work in Citizen Science: Experiments in Goal Setting and Anchoring. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW '16 Companion). ACM, New York, NY, USA, 297--300. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. A. E. Jaskot and M.S. Oey. 2013. The Origin and Optical Depth of Ionizing Radiation in the "Green Pea" Galaxies. The Astrophysical Journal 766 (2013), 91.Google ScholarGoogle ScholarCross RefCross Ref
  45. Marina Jirotka, Charlotte P. Lee, and Gary M. Olson. 2013. Supporting scientific collaboration: Methods, tools and concepts. Computer Supported Cooperative Work: CSCW: An International Journal 22, 4--6 (2013), 667--715. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Sunyoung Kim, Jennifer Mankoff, and Eric Paulos. 2013. Sensr: Evaluating a Flexible Framework for Authoring Mobile Data-collection Tools for Citizen Science. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1453--1462. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Thomas Kuhn. 2012. The Structure of Scientific Revolutions (4th ed.). University of Chicago Press, Chicago, IL.Google ScholarGoogle Scholar
  48. Deepak Kulkarni and Herbert A. Simon. 1988. The processes of scientific discovery: The strategy of experimentation. Cognitive Science 12, 2 (1988), 139--175.Google ScholarGoogle ScholarCross RefCross Ref
  49. Bruno Latour and Steve Woolgar. 1986. Laboratory life: the Social Construction of Scientific Facts (2nd ed.). Princeton University Press, Beverly Hills, CA.Google ScholarGoogle Scholar
  50. Edith Law, Connor Dalton, Nick Merrill, Alex Young, and Krzysztof Z. Gajos. 2013. Curio: a platform for supporting mixed-expertise crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP '13). AAAI, 1--2.Google ScholarGoogle Scholar
  51. Chris J. Lintott, Kevin Schawinski, Anze Slosar, Kate Land, Steven Bamford, Daniel Thomas, M. Jordan Raddick, Robert C. Nichol, Alex Szalay, Dan Andreescu, Phil Murray, and Jan Vandenberg. 2008. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179--1189.Google ScholarGoogle ScholarCross RefCross Ref
  52. Christopher S. Lowry and Michael N. Fienen. 2013. CrowdHydrology: Crowdsourcing hydrologic data and engaging citizen scientists. GroundWater 51, 1 (2013), 151--156.Google ScholarGoogle ScholarCross RefCross Ref
  53. Marder, Jenny and Fritz, Mike. (????).Google ScholarGoogle Scholar
  54. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (2012), 1--23.Google ScholarGoogle ScholarCross RefCross Ref
  55. R. Merton, E. Barber, and J. Shulman. 2006. The Travels and Adventures of Serendipity: A Study in Sociological Semantics and the Sociology of Science. Princeton University Press.Google ScholarGoogle Scholar
  56. New York University. 2016. Project Noah. (2016). Retrieved January 15, 2016 from http://www.projectnoah.org.Google ScholarGoogle Scholar
  57. Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos. 2011. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 1--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Gary Olson, Ann Zimmerman, Nathan Bos, and William Wulf. 2008. Scientific Collaboration on the Internet. MIT Press, Cambridge, MA. Google ScholarGoogle Scholar
  59. Gary M. Olson, Daniel E. Atkins, Robert Clauer, Thomas A. Finholt, Farnam Jahanian, Timothy L. Killeen, Atul Prakash, and Terry Weymouth. 1998. The Upper Atmospheric Research Collaboratory (UARC). interactions 5, 3 (May 1998), 48--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Johan Oomen and Lora Aroyo. 2011. Crowdsourcing in the Cultural Heritage Domain : Opportunities and Challenges. C&T '11 Proceedings of the 5th International Conference on Communities and Technologies July (2011), 138--149. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Michael Polanyi. 1967. The Tacit Dimension. Routledge and Kegan Paul, London, UK.Google ScholarGoogle Scholar
  62. J. Preece and B. Schneiderman. 2009. The Reader-to-Leader Framework: Motivating Technology-Mediated Social Participation. AIS Transactions on Human-Computer Interaction 1, 1 (2009), 13--32.Google ScholarGoogle ScholarCross RefCross Ref
  63. Nancy Proctor. 2013. Crowdsourcing--An Introduction: From Public Goods to Public Good. Curator: The Museum Journal 56, 1 (January 2013), 105--106.Google ScholarGoogle Scholar
  64. Danial Qaurooni, Ali Ghazinejad, Inna Kouper, and Hamid Ekbia. 2016. Citizens for Science and Science for Citizens: The View from Participatory Design. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1822--1826. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. M. Jordan Raddick, Georgia Bracey, Pamela L. Gay, Chris J. Lintott, Phil Murray, Kevin Schawinski, Alexander S. Szalay, and Jan Vandenberg. 2010. Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers. Astronomy Education Review 9, 1 (2010), 010103.Google ScholarGoogle ScholarCross RefCross Ref
  66. Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW '15). ACM, New York, NY, USA, 1364--1378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 75--85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Hauke Riesch and Clive Potter. 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Understanding of Science 23, 1 (2014), 107--120.Google ScholarGoogle ScholarCross RefCross Ref
  69. Dana Rotman, Jenny Preece, Jen Hammock, Kezee Procita, Derek Hansen, Cynthia Parr, Darcy Lewis, and David Jacobs. 2012. Dynamic Changes in Motivation in Collaborative Citizen-science Projects. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 217--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. Sharman Apt Russell. 2014. Diary of a Citizen Scientist. Oregon State University Press, Corvallis, OR.Google ScholarGoogle Scholar
  71. SciFabric. 2016a. CrowdCrafting.org Website. (2016). Retrieved January 15, 2016 from http://crowdcrafting.org.Google ScholarGoogle Scholar
  72. SciFabric. 2016b. Pybossa. (2016). Retrieved January 15, 2016 from http://www.pybossa.com.Google ScholarGoogle Scholar
  73. SciStarter. 2016. Project Finder. (2016). Retrieved January 15, 2016 from http://scistarter.com/finder.Google ScholarGoogle Scholar
  74. S. Shapin. 1995. Here and Everywhere: Sociology of Scientific Knowledge. Annual Review of Sociology 21, 1 (1995), 289--321.Google ScholarGoogle ScholarCross RefCross Ref
  75. Jennifer Shirk. 2014. Push the Edge of Science Forward: Expanding Considerations of Expertise Through Scientists' Citizen Science Work in Conservation. Ph.D. Dissertation. Cornell University, Ithaca, NY.Google ScholarGoogle Scholar
  76. Jennifer L. Shirk, Heidi L. Ballard, Candie C. Wilderman, Tina Phillips, Andrea Wiggins, Rebecca Jordan, Ellen McCallie, Matthew Minarchek, Bruce V. Lewenstein, Marianne E. Krasny, and Rick Bonney. 2012. Public participation in scientific research: A framework for deliberate design. Ecology and Society 17, 2 (2012).Google ScholarGoogle Scholar
  77. Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in ecology & evolution 24, 9 (2009), 467--71.Google ScholarGoogle Scholar
  78. Robert Simpson, Kevin R. Page, and David De Roure. 2014. Zooniverse: Observing the World's Largest Citizen Science Platform. In Proceedings of the 23rd International Conference on World Wide Web (WWW '14 Companion). ACM, New York, NY, USA, 1049--1054. Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. Ryo Suzuki, Niloufar Salehi, Michelle S. Lam, Juan C. Marroquin, and Michael S. Bernstein. 2016. Atelier: Repurposing Expert Crowdsourcing Tasks As Micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2645--2656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  80. Carol Tenopir, Suzie Allard, Kimberly Douglass, Arsev Umur Aydinoglu, Lei Wu, Eleanor Read, Maribeth Manoff, and Mike Frame. 2011. Data sharing by scientists: Practices and perceptions. PLoS ONE 6, 6 (2011).Google ScholarGoogle Scholar
  81. Ramine Tinati, Max Van Kleek, Elena Simperl, Markus Luczak-Rösch, Robert Simpson, and Nigel Shadbolt. 2015. Designing for Citizen Data Analysis: A Cross-Sectional Case Study of a Multi-Domain Citizen Science Platform. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 4069--4078. Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. Sharon Traweek. 1988. Beamtimes and Lifetimes: the World of High Energy Physicists. Harvard University Press, Cambridge, MA.Google ScholarGoogle Scholar
  83. Michael L Tushman and David A Nadler. 1978. Information Processing as an Integrating Concept in Organizational Design. Source: The Academy of Management Review 3, 3 (1978), 613--624.Google ScholarGoogle ScholarCross RefCross Ref
  84. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: heuristics and biases. Science 185, 4157 (1974), 1124--1131.Google ScholarGoogle Scholar
  85. University College of London. 2016. Transcribe Bentham: A Participatory Initiative. (2016). Retrieved January 15, 2016 from http://blogs.ucl.ac.uk/transcribe-bentham/.Google ScholarGoogle Scholar
  86. Andrew J Westphal, Anna L Butterworth, Christopher J Snead, Nahide Craig, David Anderson, Steven M Jones, Donald E Brownlee, Richard Farnsworth, and Michael E Zolensky. 2005. Stardust@home: A Massively Distributed Public Search for Interstellar Dust in the Stardust Interstellar Dust Collector. In Lunar and Planetary Science XXXVI. http://ntrs.nasa.gov/search.jsp?R=20050180792Google ScholarGoogle Scholar
  87. Andrea Wiggins. 2012. Crowdsourcing Scientific Work: A Comparative Study of Technologies, Processes, and Outcomes in Citizen Science. Ph.D. Dissertation. Syracuse University, Syracuse, NY.Google ScholarGoogle Scholar
  88. Andrea Wiggins and Kevin Crowston. 2011. From Conservation to Crowdsourcing: A Typology of Citizen Science. In Proceedings of the 44th Hawaii International Conference on System Sciences (HICSS '11). IEEE Computer Society, Washington, DC, USA, 1--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  89. Andrea Wiggins and Kevin Crowston. 2012. Goals and Tasks: Two Typologies of Citizen Science Projects. In Proceedings of the 44th Hawaii International Conference on System Sciences (HICSS '12). IEEE Computer Society, Washington, DC, USA, 3426--3435. Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. A. Wiggins, G. Newman, R.D. Stevenson, and K. Crowston. 2011. Mechanisms for Data Quality and Validation in Citizen Science. In IEEE seventh International Conference on e-Science Workshops (eScienceW '11). 14--19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. Alex C Williams, John F Wallin, Haoyu Yu, Marco Perale, Hyrum D Carroll, Anne-Francoise Lamblin, Lucy Fortson, Dirk Obbink, Chris J Lintott, and James H Brusuelas. 2014. A computational pipeline for crowdsourced transcriptions of Ancient Greek papyrus fragments. In Proceedings of 2014 IEEE International Conference on Big Data. 100--105.Google ScholarGoogle ScholarCross RefCross Ref
  92. Chris Wood, Brian Sullivan, Marshall Iliff, Daniel Fink, and Steve Kelling. 2011. eBird: Engaging birders in science and conservation. PLoS Biology 9, 12 (2011).Google ScholarGoogle Scholar
  93. A Zimmerman. 2008. New knowledge from old data The role of standards in the sharing and reuse of ecological data. Science Technology & Human Values 33, 5 (2008), 631--652.Google ScholarGoogle ScholarCross RefCross Ref
  94. Ann Zimmerman, Bos Nathan, Judy Olson, and Gary Olson. 2009. e-Research: Transformation in Scholarly Practice. Routledge, Chapter The Promise of Data in e-Research: Many Challenges, Multiple Solutions, Diverse Outcomes, 222--239.Google ScholarGoogle Scholar
  95. Zooniverse. 2016a. AnnoTate. (2016). Retrieved January 15, 2016 from https://anno.tate.org.uk/.Google ScholarGoogle Scholar
  96. Zooniverse. 2016b. Old Weather Project. (2016). Retrieved January 15, 2016 from http://www.oldweather.org/.Google ScholarGoogle Scholar
  97. Zooniverse. 2016c. Operation War Diary. (2016). Retrieved January 15, 2016 from http://www.operationwardiary.org/.Google ScholarGoogle Scholar

Index Terms

  1. Crowdsourcing as a Tool for Research: Implications of Uncertainty

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
      February 2017
      2556 pages
      ISBN:9781450343350
      DOI:10.1145/2998181

      Copyright © 2017 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 25 February 2017

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      CSCW '17 Paper Acceptance Rate183of530submissions,35%Overall Acceptance Rate2,235of8,521submissions,26%

      Upcoming Conference

      CSCW '24

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader