AI Learning

Decentralized Governance to Optimize Human Output Datasets for AI Learning

`` WulfKaal,Ph.D.1 ``

Abstract

`` TheevolutionofAIdependsonupgradablequalitydatasets.Data isthefoundationonwhichAIalgorithmslearnandmake predictions.High-quality,diverse,andlabeleddatasetsare crucialfortrainingAImodelseffectively.Theavailabilityof qualitydataplaysasignificantroleindeterminingthesuccess andimpactofAIindisruptedindustries. ``

`` TheAILearningEcosystem(ALE)facilitatesamicrotask ecosystemforAIlearning.ALEusesitsprovenandtested decentralizedgovernanceecosystemtoprovidehigh-quality diversedatasetsforAIlearningviagamifiedmicro-taskwork. Throughitstestingenvironmentintheindustry-leadingCode ReviewDAO(CRDAO),ALEdistinguishesitselffromcompetitors throughunparalleleddecentralizedgovernanceoptimizationthat minimizesmicro-taskworkduplicationincentralizedsystemsand allowsgamifiedmicro-taskworktoscalehigh-qualitydiverse datasetsforAIlearning. ``

`` KeyWords:ArtificialIntelligence,LargeLanguageModels, Dataset,MicroTaskWork,Gamification,QualityControls, DecentralizedAutonomousOrganization,TokenModels, Cryptocurrencies,FeedbackEffects,EmergingTechnology,Tokens, Blockchain,DistributedLedgerTechnology,CodeAssurances ``

`` JELCategories:K20,K23,K32,L43,L5,O31,O32 ``

1 The Author is grateful to many members of ALE and the open source community who provided valuable feedback.

Table of Contents

Background..................................................................................................................................3

AI for Human Evolution............................................................................................................3

AI Business Model Disruption and Scaling..............................................................................4

Pain Point Data Quality............................................................................................................6

Total Addressable Market........................................................................................................ 8

Disrupted Industries...........................................................................................................9

Data Providers................................................................................................................. 14

Gamification to enhance AI Data Quality................................................................................ 15

Freelance Micro Task Market Growth....................................................................................16

Micro Task Work for AI Learning............................................................................................18

Shortcomings in Legacy Micro Task Market.......................................................................... 20

Lack of Scaling Solutions in Centralized Micro Task Market............................................21

Overpricing.................................................................................................................21

Duplication of Work....................................................................................................22

No Access for Workers.............................................................................................. 23

Gamification to Enhance Micro Task Work............................................................................ 23

Case Study - Code Review Platform........................................................................................26

The Code Review Platform is operated by an affiliated entity and served as a testing<br>environment in the past. It is accessible under the following URL: https://crdao.ossa.dev/.. 26

Community Audit for Code Review........................................................................................26

AI Learning Ecosystem Platform............................................................................................. 46

Scaling the Code Review Platform with Micro Task Work..................................................... 46

Removing Cost of Micro Task Work Duplication....................................................................46

Transaction Cost Minimization of Micro Task Work............................................................... 49

Gamification of Micro Task Work........................................................................................... 49

Background

AI for Human Evolution

`` ArtificialIntelligence(AI)epitomizeshumanevolution.Based onfindingsassociatedwithhumanbrainfunctionality,deep learninginAIsystemsisamethodofproducingAIcomponents thatcanprocessdatasimilartohowthehumanbrainwould processdata.ThissupportstheAI’srecognitionofcomplex patternsindatasets,includinginpictures,textelements,and sounds.Theanalysisofsuchdatasetsindeeplearningallows theAItogenerateinsights,analysis,andpredictionsfora varietyofhumangeneratedoutcomes.ThroughdeeplearningAI canautomatesequencesoftasksthatotherwisewouldrequire humanintelligence.Examplesmayincludetheanalysisandcausal recognitionofimagesortranscribingasoundfileintotext, amongothers. ``

`` In2024,AIdeeplearningispossiblethroughsocalledLarge languagemodels(LLM).LLMsaretrainedonlargedatasets.For thispurpose,neuralnetworks,throughtheirencodersand decodersextracthumanlevelmeaningsfromasequenceofdata includingtextandcreateassociatedrelationshipsbetweenwords andphrasesinthetextthatiscontainedinthelargedatasets. ``

`` So-calledtransformerneuralnetworkarchitectureenablesthe utilizationoflargedatasetswithuptohundredsofbillionsof parameters.Whiletransformerneuralnetworkarchitecturecan utilizehugedatasetsthatareoftengeneratedfrominternet data,withoftenover50billionwebpages,andWikipedia,with itsevergrowingamountofpages,dataqualitycontinuestobea hugeissueforLLMs. ``

AI Business Model Disruption and Scaling

`` DeeplearningAIhasunparalleledusecasesthatallowan industry-widescalingofnewbusinessmodelsbuiltonAI, includinginelectricvehicles,legal,administrative, manufacturing,aerospace,electronics,medicalresearch,and ``

`` manyotherindustriesandfields. ``

`` AIhasthepotentialtodisruptexistingbusinessmodelsand enablethecreationofnewandinnovativebusinessesinseveral ways: ``

`` 1.Automationandefficiency:AIcanautomaterepetitiveand mundanetasks,leadingtoincreasedefficiencyandcostsavings. Thisdisruptioncaneliminatetheneedforcertainjobrolesor transformthem,allowingbusinessestoallocateresourcesmore effectivelyandfocusonhigher-valueactivities. ``

`` 2.Enhanceddecision-making:AIalgorithmscananalyzevast amountsofdataquicklyandaccurately,providingvaluable insightsfordecision-making.Thisdisruptionempowers businessestomakedata-drivendecisions,optimizeprocesses, andimproveoutcomes. ``

`` 3.Personalizationandcustomerexperience:AIenables businessestopersonalizetheirproducts,services,andcustomer experiencesbasedonindividualpreferencesandbehavior.By leveragingAIalgorithms,businessescancreatetailored recommendations,targetedmarketingcampaigns,andpersonalized interactions,enhancingcustomersatisfactionandloyalty. ``

`` 4.Predictiveanalytics:AIalgorithmscanpredictfuture trends,customerbehavior,andmarketdynamicsbyanalyzing historicaldata.Thisdisruptionallowsbusinessestoanticipate demand,optimizeinventory,mitigaterisks,andmakeproactive decisions,givingthemacompetitiveadvantage. ``

`` 5.Newbusinessmodelsandservices:AIopensdoorstoentirely ``

`` newbusinessmodelsandservices.Forexample,AI-powered platformscanconnectconsumersandprovidersdirectly, disruptingtraditionalintermediaries.Additionally,AIcan enablethecreationofinnovativeproductsandservicesthat werenotpossiblebefore,suchasautonomousvehicles,virtual assistants,orpersonalizedhealthcaresolutions. ``

`` 6.Scalabilityandscalability:AIsystemscanscaleefficiently andhandlelargevolumesofdataandinteractions,makingit easierforbusinessestoexpandtheiroperationsrapidly.This disruptionallowsbusinessestoreachbroadermarketsandserve alargercustomerbasewithoutsignificantinfrastructure investments. ``

Pain Point Data Quality

`` ThegrowthandsuccessofAIarehighlydependentonquality datasets.DataisthefuelthatpowersAIalgorithms,and withouthigh-quality,relevant,anddiversedata,the performanceandaccuracyofAIsystemscanbecompromised. ``

`` Whileitisimportanttonotethatdataqualityaloneisnot sufficientforAIgrowthanditsdisruptionandreinventionof businessmodels,thealgorithms,computingpower,andexpertise ofdevelopersalsoplaysignificantroles.Nonetheless,access toqualitydatasetsisafundamentalrequirementforthe developmentandadvancementofAItechnologies. ``

`` Thekeyreasonswhyhighqualitydatasetsarecrucialforthe AIevolutionincludebutarenotlimitedtothefollowing: ``

1. Training AI models: AI systems learn from data through a

`` processcalledtraining.Duringtraining,AIalgorithmsanalyze andextractpatternsfromlargedatasetstomakepredictionsor performspecifictasks.Thequalityofthetrainingdata directlyimpactstheAImodel'sabilitytolearnandmake accuratepredictions. ``

`` 2.Biasreduction:High-qualitydatasetshelpinreducingbias ``

`` withinAIsystems.Biaseddatacanleadtobiasedoutcomes, perpetuatingsocialorculturalprejudices.Byensuringdiverse andrepresentativedatasets,AIdeveloperscanworktowards minimizingbiasandcreatingfairerAIsystems. ``

`` 3.Generalizationandadaptability:QualitydatasetshelpAI modelsgeneralizetheirlearningtonewsituationsandadaptto changingenvironments.Adiverseandcomprehensivedataset allowsAIsystemstoencounterawiderangeofscenarios, leadingtobetterperformanceinreal-worldapplications. ``

`` 4.Robustnessandreliability:AImodelstrainedonhigh-quality datasetstendtobemorerobustandreliable.Theycanhandle edgecases,outliers,andunexpectedinputswithgreater accuracy,improvingtheoverallperformanceanduserexperience. ``

`` In2024,AIdevelopmentcompaniesarebeginningtousesmaller datasets2forthedevelopmentofLLMs.Thisisdoneinaneffort toincreasemodelaccuracyandpreventoverfittinginmachine learningmodels,which,inturn,allowsthenewmachinelearning modelstoproducebetterresultsonunseendata.However,with smalldatasetsinLLMs,theriskofoverfittingalsorises, especiallywithcomplexmodels.Therefore,LLMdevelopershave toturntoregularizationinanefforttoaddressoverfittingof themodelwiththetrainingdata. ``

`` BecauseofthetrendtowardssmallerdatasetsinLLMs,data qualityisacrucialpainpointinsmallerdatasets.Inaddition tobalancingandnormalizingthedata,LLMdevelopersfocuson adequatemodelvalidationthroughtechniquesincludingdropouts inneuralnetworks,pruningdecisiontrees,and cross-validation.Alloftheseeffortsaredevotedtogenerating bettertrainingdatasetsasthequalityofLLMtrainingdatacan breakLLMresultsorimprovethemsignificantly. ``

Total Addressable Market

`` Industriesacrosstheboardcanbesignificantlyaffectedbythe availabilityofhigh-qualityAItrainingdatasets.Some industriesmayexperienceamorepronouncedimpactduetothe ``

2 https://towardsdatascience.com/is-small-data-the-next-big-thing-in-data-science-9acc7f24907f

`` natureoftheiroperationsandthepotentialforAI-driven ``

`` transformations. ``

Disrupted Industries

`` Estimatingthetotaladdressablemarket(TAM)forindustries impactedbytechnologicaladvancements,particularlyartificial intelligence(AI),isamultifacetedtask.Itrequiresanalyzing varioussectors,eachwithitsownmarketsizeandgrowth trajectory.ThedynamicnatureofAI'sinfluencecomplicatesthe efforttopinpointanexactmarketvalue.Nonetheless,the aggregatepotentialmarketsizeisundeniablyvast. ``

`` TocontextualizetheTAMforindustriesundergoingdisruption, ``

`` considerthefollowingsectors: ``

1. Healthcare:

`` AIapplicationsinhealthcare,suchasdiseasediagnosis,drug discovery,personalizedmedicine,andpatientmonitoring,depend onhigh-qualitydatasetsforaccuracy.Theseapplicationsare poisedtoenhancemedicaldecision-makingandpatientcare significantly.AccordingtotheWorldHealthOrganization,the globalhealthcaremarketwasestimatedat$8.45trillionin2020 ``

`` andisprojectedtogrowto$11.91trillionby2025, ``

`` highlightingAI'scriticalroleinthissector'sevolution.3 ``

2. Finance:

`` Thefinancesectorreliesondataforfunctionsincludingrisk assessment,frauddetection,andalgorithmictrading.AImodels thatprocessthisdatacanuncoverpatternsandpredictmarket movements,therebyfacilitatingsmarterinvestmentstrategies andcustomerserviceimprovements.Theglobalfinancialservices market'svaluestoodatabout$19.7trillionin2020,withAI expectedtotransformbanking,insurance,andassetmanagement.4 ``

3. Retail and E-commerce:

`` Inretailande-commerce,AI-driventoolsforpersonalization, demandforecasting,andinventorymanagementrelyondetailed dataanalysis.Thesetoolsenablebusinessestotailortheir offeringsandoptimizeoperations.Theglobalretailmarket's valuationexceeded$22trillionin2020,andAI'scontributions aresettoredefinetheindustrybyboostinggrowthand ``

`` enhancingtheconsumerexperience.5 ``

3 "Global Spending on Health: A World in Transition," World Health Organization, 2021.

4 "Global Financial Markets: An Overview," International Monetary Fund, 2020.

5 "The State of the Global Retail Market," World Retail Congress, 2020.

4. Manufacturing and Logistics:

`` AIenhancesmanufacturingandlogisticsthroughimproved efficiency,qualitycontrol,andsupplychainmanagement.The globalmanufacturingmarketwasvaluedataround$39.6trillion in2020.AIapplicationsinthissectorpromisesignificantcost reductionsandperformanceenhancements.6 ``

5. Transportation:

`` AI'spotentialintransportation,includingautonomousvehicles andtrafficmanagement,isconsiderable.Theglobal transportationservicesmarketwasvaluedatapproximately$4.3 trillionin2020.AsAItechnologiessuchasautonomousdriving androuteoptimizationadvance,theTAMforAIintransportation isexpectedtowitnesssubstantialgrowth.7 ``

6. Customer Service:

`` ThetransformationofcustomerservicebyAI,throughchatbots andvirtualassistants,underscoresthetechnology'svaluein enhancingcustomerinteractions.Thecustomerservicesector, valuedatabout$55billionin2020,willseeitsTAMforAI ``

6 "Global Manufacturing Outlook," United Nations Industrial Development Organization, 2021.

7 "Global Transportation Market Analysis," International Transport Forum, 2020.

`` expandasbusinessesincreasinglyadoptthesesolutionsto ``

`` improveefficiencyandcustomersatisfaction.8 ``

`` Thesesectorsexemplifythesignificant,albeitchallengingto quantify,TAMforAIacrossindustries.Theongoingintegration ofAIintotheseareaspromisesnotonlytodrivemarketgrowth butalsotocatalyzeprofoundchangesinhowservicesare deliveredandconsumedworldwide. ``

`` Theimpactofhigh-qualityAItrainingdatasetsisnotlimited totheseabovementionedindustries.AIhasthepotentialto disruptandtransformvarioussectors,includingeducation, energy,agriculture,entertainment,andmore.Theavailability ofqualitydatasetsisacatalystforAI-drivenadvancements andinnovationacrossindustries. ``

`` PleasenotethattheseTAMfiguresareapproximateandsubject tovariousfactorssuchasmarketdynamics,technological advancements,andglobaleconomicconditions.Thegrowth potentialofAIintheseindustriesissignificantandwill continuetoevolveasthetechnologymaturesandadoption increases. ``

8 "Global Customer Service Market Report," Customer Service Institute, 2020.

`` However,acquiringandcuratinghigh-quality,diverse,and labeleddatasetscanbeachallenge.Industriesthatcanaccess largevolumesofstructuredandlabeleddata,suchasfinance andhealthcare,mayhaveaheadstartinleveragingAI.Onthe otherhand,industriesliketransportationandcustomerservice mayfacehurdlesinobtainingqualitydataduetoprivacy concerns,datafragmentation,andtheneedforextensive labelingefforts.Overcomingthesechallengesandensuring accesstoqualitydataisessentialformaximizingthepotential TAMofAIinthesesectors. ``

`` Despitethesechallenges,theTAMforAIintransportationand customerserviceissubstantial.AsAItechnologiesmature,data availabilityimproves,andcompaniesinvestinAI-driven solutions,theTAMfortheseindustriesisexpectedtoincrease significantly.ThecombinedTAMofalldisruptedindustries, includinghealthcare,finance,retail,manufacturing,logistics, transportation,andcustomerservice,islikelytobeinthe hundredsoftrillionsofdollars,makingitahighlylucrative marketforAI-driveninnovation.AddingtheTAMofother affectedsectorslikeeducation,energy,agriculture,and entertainment,thetotaladdressablemarketforalldisrupted ``

`` industriescombinedisinthehigherhundredsoftrillionsof ``

`` dollars. ``

Data Providers

`` DeterminingtheexactTAMfordataproviderswhoproduce high-qualitydiversedatasetsforAIlearningischallengingas itdependsonvariousfactorssuchasmarketdemand,pricing, andcompetition.However,itissafetosaythattheTAMfor suchdataprovidersissignificantandgrowingrapidly. ``

`` AsAIcontinuestoadvanceandbecomemoreprevalentacross industries,theneedforqualitytrainingdatabecomes paramount.Dataprovidersthatcandeliverlabeled,diverse,and reliabledatasetstotrainAImodelseffectivelywillhavea ``

`` significantmarketopportunity. ``

`` TheTAMfordataprovidersinAIlearningextendsacross multiplesectors,includinghealthcare,finance,retail, manufacturing,logistics,transportation,customerservice, education,energy,agriculture,andentertainment,amongothers. Eachoftheseindustriesrequiresspecificdatasetstailoredto ``

`` theiruniqueneedsandusecases. ``

`` Furthermore,withtheincreasingadoptionofAIbybothlarge enterprisesandsmallerbusinesses,thedemandforhigh-quality dataisexpectedtosurge.Startups,establishedcompanies,and evenAIserviceproviderswillrelyondataproviderstoaccess thenecessarydatasetsfortheirAIinitiatives. ``

`` ConsideringthepotentialTAMofthedisruptedindustries mentionedearlier,whichisestimatedtobeinthehundredsof trillionsofdollars,theTAMfordataproviderswhoproduce high-qualitydiversedatasetsforAIlearningislikelytobe substantial.AsAIbecomesmoreintegraltobusinessoperations anddecision-makingprocesses,theimportanceofquality trainingdatawillonlygrow,furtherexpandingtheTAMfordata providersinthisspace. ``

`` GamificationtoenhanceAIDataQuality ``

`` TheAILearningEcosystem(ALE)usesitsprovenandtested decentralizedgovernanceecosystemtoprovidehigh-quality ``

`` diversedatasetsforAIlearningviamicro-taskwork. ``

`` ThroughitstestingenvironmentintheexistingCodeReviewDAO, ``

`` ALEdistinguishesitselffromcompetitorsthrough ``

`` industry-leadingdecentralizedgovernanceoptimizationthat ``

`` minimizesmicro-taskworkduplicationincentralizedsystemsand allowsmicrotaskworktoscaleforAIdatasetgeneration. ``

Freelance Micro Task Market Growth

`` Inthecurrentfreelancermarket,leadingcentralizedplatforms Fiverr,Taskrabbit,Upwork,CrowdflowerandMechanicalTurkhave theirworkforcesdistributedthroughouttheworld.MicroTasks freelanceworkisbecomingincreasinglyimportantinthe freelancemarket. ``

`` Microtasksaredefinedassmalltasksthatrequirehuman judgment,canbecompletedbyhumansindependentlyoverthe internet,andarepartofalargerunifiedproject.Becauseof thenecessityofhumanjudgmentthatcannotcurrentlybe replacedbymachines/computers,microtasksenableorganizations tobuildproductsorcreateoutcomesthatcannotbesynthesized bymachines/computersalone.Forexample,theChinesegovernment uses2millionmicrotaskworkerstoaidincensoringthe internet.Internetcompanies,suchasGoogle,Facebook,Twitter, Ebay,andLinkedinoptimizetheirproductionreadysolutionsand enhancetheirAImodeltrainingwithmicrotaskworkers.Large scaledistributionsofmachinelearningresearchers,among ``

`` others,gatherstructuredlabeleddataforartificial intelligence(AI)trainingpurposesthroughtheuseofmicro ``

`` taskworkers. ``

`` Billionsofmicrotasksarecompletedeachyearandthedemand formicrotaskworkersisincreasingconsistently.Inanattempt tocapitalizeonthegrowingdemandformicrotaskwork,Amazon createdAmazonMechanicalTurk(MTurk)in2005.MTurkisan onlinemarketplacethatallowsrequesterstopayworkersfor performingmicrotasksonline,thuscrowdsourcingdata collection.TheWorldBankconcludedina2015report,thatthe largestcrowdsourceddatacollectionplatformsAmazonMechanical TurkandCrowdFlower,aventurebackedcompanythatraisedover $58millionandisfocusedonenrichingdatausedforAI,would quadrupletheirrevenuefrom2013to2016. ``

`` Theincreasingdemandformicrotaskworkersthatcameinitially fromdatascientistsandotheracademicsisfurtherincreasedby thegrowthofartificialintelligence(AI)andtheincreasing scopeandscaleofAIapplicationsinabroadrangeof industries.Fortune500techcompaniesincludingAmazon,Apple, GoogleAlphabet,Twitter,andFacebookuseAIandmachine learningtoimprovetheirservicesandcutcosts,thus increasingprofitabilityexponentially.Accordingtosome ``

`` estimates,Fortune500techcompaniesspentbetween$20and$30 ``

`` billiononthedevelopmentandenhancementoftheirAIsystems in2016.Morerecentdatafor2017suggeststhatthistrendis continuingandincreasing.AIisalreadyplayingasignificant roleinconsumerexpectationsandtechcompanieshavethebest possiblesetofincentivestokeepinvestinginAItofulfill ``

`` suchexpectationsanddevelopnewproductsatmarginalcost. ``

Micro Task Work for AI Learning

`` Microtaskworkplaysacrucialroleinproducinghigh-quality anddiversedatasetsforAIlearning.Itinvolvesbreakingdown complextasksintosmaller,moremanageabletasksthatcanbe completedbyalargenumberofhumanworkers.Theseworkers, oftenreferredtoascrowdworkersormicrotaskers,perform thesetasksinexchangeformonetarycompensation. ``

`` Thefollowingexamplesshowhowmicrotaskworkcontributesto thecreationofhigh-qualitydatasetsforAIlearning: ``

`` 1.DataAnnotation:Microtaskworkiscommonlyusedfordata annotationtasks,wherecrowdworkerslabelandannotatedata accordingtospecificcriteria.Forexample,inimage ``

`` recognition,crowdworkersmayannotateobjectsordrawbounding boxesaroundthemtotrainAImodels.Thisannotationprocess helpscreatelabeleddatasetsthatserveasgroundtruthfor trainingAIalgorithms. ``

`` 2.DataValidation:Crowdworkersalsoplayacriticalrolein validatingthequalityandaccuracyoflabeleddata.Theycan reviewandverifyannotationsmadebyotherworkers,ensuring consistencyandreducingerrors.Thisvalidationstephelps ``

`` maintaintheintegrityandreliabilityofthedataset. ``

3. Data Augmentation: Micro task work can be utilized to

`` generatediversedatabycreatingvariationsoraugmentationsof existingdatasets.Crowdworkerscanperformtaskslikeimage manipulation,textparaphrasing,oraudiosynthesistoincrease ``

`` thediversityofthetrainingdata.ThishelpsAImodels generalizebetterandperformwellonawiderrangeof ``

`` real-worldscenarios. ``

`` 4.DataCleaning:Crowdworkerscanassistincleaningand refiningdatasetsbyidentifyingandrectifyingerrors,removing duplicates,orstandardizingdataformats.Thisensuresthe datasetisofhighquality,reducingnoiseandimprovingthe performanceofAImodelsduringtrainingandinference. ``

`` Microtaskworkplatforms,suchasAmazonMechanicalTurk, FigureEight(nowAppen),andScaleAI,providethe infrastructuretodistributethesetaskstoalargepoolof workers,managetheircontributions,andensurequalitycontrol. ``

`` Overall,microtaskworkenablestheefficientandscalable productionofhigh-quality,diversedatasetsforAIlearning.It leverageshumanintelligencetohandletasksthatare challengingforcurrentAIsystemsandcontributestothe continuousimprovementandadvancementofAItechnologies. ``

Shortcomings in Legacy Micro Task Market

`` Theevolution,improvements,andgrowthofAIiscorrelatedwith theevolution,improvements,andgrowthinmicrotaskwork.AI usessupervisedandunsupervisedaswellasreinforcement machinelearning.Becauseunsupervisedandreinforcement learningaremuchmorecomplexthansupervisedlearning, supervisedlearningistodatemorecommonandmorereliedupon forAIdevelopment.Whilethismaychangeasunsupervisedand reinforcementlearningevolve,currently,supervisedlearning ``

`` dependsonlabeleddatathatisproducedviamicrotaskwork. ThemappingfunctionofthesupervisedAIlearningprocess necessitatestheanalysisoflabeledinputvariablesxand correspondingoutputvariablesy.Inthesupervisedlearning trainingphase,theAIneuralnetworkexaminesthetraining datasetoflabeledxinputdatatolearntoclassifytheinput dataidealistically.Thehigherthequalityandquantityofsuch labeleddatasetsthebettertheAIneuralnetwork’slearning algorithmduringthesupervisedtrainingprocess.Accordingly, theevolution,improvements,andgrowthofAIiscorrelatedwith theevolution,improvements,andgrowthinmicrotaskswork.But alas,microtasksplatformsystemsaresubjecttosignificant limitationsthatinhibittheevolutionofAI. ``

`` LackofScalingSolutionsinCentralizedMicroTaskMarket ``

Overpricing

`` Theexistingcentralizedmarketplacesformicrotaskworkcannot adequatelyfulfilltheincreasingdemandforhighqualitymicro taskworkforAIlabeledtrainingdatasets.Firstandforemost, thecoststructureformicrotaskworkincentralizedsystems thatnecessitateintermediationresultsinsignificant overpricingwithoutbenefitingthemicrotaskworkersdirectly. Thecoststructuresuboptimalitycanbetracedbacktoseveral factors.Alltoohumanshortcomingsofmicrotaskworks,suchas ``

`` limitedattentionspan,irrationality,andinaccuraciesresult inverificationrequirementsformicrotaskwork.However, manualverificationofmicrotaskworkissubjecttothesame humanlimitations. ``

Duplication of Work

`` Inanattempttoensurequalityofresultsandminimizethe impactofthehumanlimitationsoftheirworkers,requestersin centralizedmicrotaskstructuressetupteamsofupto15 workerstoperformthesametaskinanefforttoforma consensus.Themultiplicationofworkinherentinthisprocess significantlyincreasesthecostofmicrotaskwork.Requiring requesterstopayproportionally(e.g.upto15times)forwork perprojectresultsinwaste.Thenecessityofmultiplicationof workalsosubjectsmicrotaskworkerstolowerratesandlackof paymentincreases.Moreover,becauseunmanagedcentralizedmicro taskplatformsdonotsupplyconsumerinterfacesneededto accomplishspecifictasks,requestersofmicrotaskworkare forcedtoeitherbuildtheirowntoolsorpaylargefeesto startupsthathopetocapturetheenterprisemarket.Both optionsarenecessaryincurrentcentralizedsystemsbutalso resultinunderutilizationofresources. ``

No Access for Workers

`` Mostimportantly,thecirca38%ofthelaborpoolthatis unbankedbutskilleddoesnotcurrentlyhaveaccesstothe centralizedmicrotaskmarketplaces.Withoutabankaccount, workerscannotcontributeandprofitfromtheexisting centralizedmicrotaskmarketplace.Furthermore,evenforthose inthelaborpoolwhodohavebankaccounts,micro-taskworkis oftenassociatedwithpracticalproblemssuchashighfeesof intermediaryfinancialinstitutions,lostorotherwiseaffected payments,lostchecks,amongotherissues.Finally,micro workersincentralizedsystemsarefacedwithinvasive,privacy challenging,timeconsuming,andunclearsignupandapproval processesthatcreatemarketentrybarriersformicrotask workers. ``

Gamification to Enhance Micro Task Work

`` Thegamificationofmicrotaskworkcanhaveseveraleffectson theprocessofproducinghigh-quality,diversedatasetsforAI learning.Gamificationreferstotheapplicationofgamedesign principlesandmechanicstonon-gamecontexts,suchasmicro taskwork. ``

`` Gamificationcanimpacttheprocessofproducinghigh-quality, diversedatasetsforAIlearninginthefollowingways: ``

`` 1.IncreasedEngagement:Gamificationtechniques,likeadding points,levels,leaderboards,orrewards,allofwhichmaybe utilizedexclusivelywithintheplatform,canenhancethe engagementofcrowdworkers.Byintroducingacompetitiveor rewardingelement,gamificationmotivatesworkerstoactively participateandcompletetasksmoreefficiently.Thisincreased engagementcanleadtohigher-qualityoutputsasworkersare moreinvestedintheprocess. ``

`` 2.QualityControl:Gamificationcanbeutilizedtoimprovedata qualitythroughmechanismslikeconsensus-basedvotingorpeer review.Workerscanreviewandrateeachother'scontributions, earningpointsorrecognitionforaccurateandconsistentwork. Thisapproachhelpsidentifyandresolvediscrepancies,ensuring betterqualitycontrolinthedatasetcreationprocess. ``

`` 3.SkillDevelopment:Gamificationcanenableskilldevelopment ``

`` amongcrowdworkers.Byprovidingchallengesorlevelsthat progressivelyincreaseindifficulty,workerscanenhancetheir annotation,validation,orcleaningskillsovertime.Thisskill developmentcanresultinimprovedaccuracyandefficiencyin datasetcreation. ``

`` 4.CrowdworkerRetention:Gamificationelementslikebadges, achievements,orvirtualcurrencies,allofwhichmaybe utilizedexclusivelywithintheplatform,canenhance crowdworkerretention.Byrecognizingandrewardingworkersfor theircontributions,gamificationfostersasenseof accomplishmentandencouragesthemtocontinueparticipatingin microtaskwork.Thisretentioniscrucialformaintaininga consistentpoolofexperiencedandreliableworkersfordataset creation. ``

`` 5.ScalabilityandSpeed:Gamificationcanhelpexpeditethe datasetcreationprocessbyencouragingworkerstocomplete tasksmorequickly.Byintroducingtime-basedchallengesor offeringbonusesfortimelycompletion,gamificationcan increasethespeedatwhichhigh-qualitydatasetsaregenerated, enablingscalabilityforlarge-scaleAIprojects. ``

`` It'simportanttonotethatwhilegamificationcanpositively ``

`` impactengagementandproductivity,properdesignand implementationincombinationwithgamificationlogicof reputationgovernancearecrucial.Throughthecombinationof decentralizedgovernanceandgamificationofmicrotaskworkALE platformintendstoensurethatgamificationtechniquesdonot compromisethequalityandaccuracyofthedataset.Balancing ``

`` thegamificationelementswithappropriatequalitycontrol measuresviadecentralizedgovernanceisessentialtomaintain theintegrityofthedatasetcreationprocess. ``

Case Study - Code Review Platform

`` TheCodeReviewPlatformisoperatedbyanaffiliatedentityand servedasatestingenvironmentinthepast.Itisaccessible underthefollowingURL:https://crdao.ossa.dev/ ``

Community Audit for Code Review

`` ScalingtheCodeReviewPlatformoperationsasaproofpointis akeyobjectiveforALE.TheCodeReviewPlatformusesseveral keyreputationrelatedmetrics,asillustratedbelow,toscale itsoperationsovertime. ``

`` Toenablethepublictoparticipateinandcontributetothe CodeReviewPlatform,ALEisusingagamifiedreputationsystem. AnyusercanaccessandcontributetotheCodeReviewPlatform andearnreputationscoresontheplatform.Whilethecode reviewsthemselvesarerelegatedtotheexpertmembersofthe ``

`` CodeReviewPlatform,thepubliccansupporttheultimatecode reviewsviagamifiedengagements. ``

`` MechanicalTurk(asdefinedabove)functionalityisacore applicationoftheCodeReviewPlatformProtocolandtheCode ReviewPlatformecosystem.Otherusecasesandapplicationswill evolveontheCodeReviewPlatformovertime.TheCodeReview Platformarchitecturewillevolveinanysettingthatsupports theethicaladvancementofthecryptoevolution. ``

Growth Potential of the Smart Contract Industry

`` Thesmartcontractindustryhasenormousgrowthpotential. DifferentmeasureshelpassessthatgrowthincludingtheCAGR (compoundedannualgrowthrate)usedaswellasthecurrent valuationofthesmartcontractindustry.Estimatesofthesmart contractindustry’sfuturevalueby2032rangefrom1to2.5 billiondollars.9 ``

9 Varying group estimates are listed below from low to high.Group Estimates for the Smart Contract Industry’s Value by 2032:

1. SNS Insider: $1 billion, at a CAGR of 24.2%

1.1. Smart Contracts Market Size , SNS Insider, <u>https://www.snsinsider.com/reports/smart-contracts-market-1542</u> (last visited Jan. 4, 2024).

2. Verified Market Research: $1.2 billion, at a CAGR of 26.4%

2.1. Smart Contracts Market Size and Forecast , Verified Market Research, <u>https://www.verifiedmarketresearch.com/product/smart-contracts-market/</u> (last visited Jan. 4, 2024).

3. Valuates Reports: $1.4 billion, at a CAGR of 24.2%

Smart Contract Vulnerabilities Undermine Industry Growth

`` Thegrowthofthesmartcontractindustryacrossindustry vectorsisaffectedbytheattackvectorspertainingtosmart contracts.Smartcontractbugscanresultinfinancialloss, reputationalloss,increasedsmartcontractcosts,legalissues, and,inextremecases,destructionofthesmartcontract.10 ``

- 3.1. Global Smart Contracts Market Research Report , Valuates Reports, <u>https://reports.valuates.com/market-reports/QYRE-Auto-31L1599/global-smar t-contracts</u> (last visited Jan. 4, 2024).

4. Acumen Research & Consulting: $1.417 billion, at a CAGR of 22.8%

- 4.1. Smart Contracts Market Size: Global Industry, Share, Analysis, Trends and Forecast 2023 – 2032, ACUMEN RESEARCH & CONSULTING, https://www.acumenresearchandconsulting.com/smart-contracts-market (last visited Jan. 4, 2024).

5. Future Market Insights: $1.5 billion, at a CAGR of 23.5%

- 5.1. Smart Contracts Market Outlook (2022 to 2032) , Future Market Insights,

`` https://www.futuremarketinsights.com/reports/smart-contracts-market(last visitedJan.4,2024). ``

6. Allied Market Research: $2.5 billion, at a CAGR of 29.6%

- 6.1. Smart Contracts Market Research, 2032, ALLIED MARKET RESEARCH, https://www.alliedmarketresearch.com/smart-contracts-market-A144098 (last visited Jan. 4, 2024).

10 See David Balaban, Navigating The Security Challenges Of Smart Contracts , Forbes (Feb. 11, 2023, 6:33 AM),

`` https://www.forbes.com/sites/davidbalaban/2023/02/11/navigating-the-security-ch allenges-of-smart-contracts/?sh=14006afd4992.MajdSoud,GrischaLiebel& ``

`` MohammadHamdaqa,PrAIoritize:LearningtoPrioritizeSmartContractBugsand Vulnerabilities(WorkingPaper),https://arxiv.org/pdf/2308.11082.pdf.Sherman Lee,BlockchainSmartContracts:MoreTroubleThanTheyAreWorth?Forbes(Jul 10,2018,11:38PM), ``

`` https://www.forbes.com/sites/shermanlee/2018/07/10/blockchain-smart-contracts-m ore-trouble-than-they-are-worth/?sh=74b5654523a6.HaozheZhou,AminMilaniFard &AdetokunboMakanju,TheStateofEthereumSmartContractsSecurity,2 JournalofCybersecurity&Privacy358(2022).HantingChuetal.,ASurveyon SmartContractVulnerabilities:DataSources,Detection,andRepair,196 INFORMATION&SOFTWARETECHNOLOGY1(2023), ``

`` https://www.sciencedirect.com/science/article/pii/S0950584923000757.MacKenzie Sigalos,BugPuts$162MillionupforGrabs,SaysFounderofDeFiPlatform Compound,MSNBC(Oct.3,2021,2:41PM), ``

`` https://www.cnbc.com/2021/10/03/162-million-up-for-grabs-after-bug-in-defi-prot ``

`` ocol-compound-.html.TamerAbdelaziz&AquinasHobor,SmartLearningtoFind DumbContracts, ``

`` https://www.usenix.org/system/files/usenixsecurity23-abdelaziz.pdf.Fabio Grittietal.,ConfusumContractum:ConfusedDeputyVulnerabilitiesin EthereumSmartContracts, ``

`` Whilenocleardataexiststoestimatehowmuchsmartcontract ``

`` attackvectorswilllimitthegrowthofthesmartcontract industry,thesmartcontractsindustryisstillestimatedto growtoseveralbilliondollarsinthenextdecade—despitethe costofsecurityauditsandfinanciallossesfrombadactors exploitingsmartcontractsbugs.Estimatesonfinancialloss attributabletosecuritybreachesandattachvectorsofsmart contractsrangefromhundredsofmillionstobillionsof dollars.Forexample,between2016and2018,sevencybersecurity incidentsoccurredinEthereumsmartcontractsresultingin financiallossesofover$289million.11In2021alonetotal financiallossfromsmartcontractbugswasestimatedat$680 million.12Someestimatesputthecurrentglobalfinancialloss ``

`` duetosmartcontractvulnerabilitiesover6billiondollars.13 ``

`` https://www.usenix.org/system/files/usenixsecurity23-gritti.pdf.Inonestudy, researchersfoundthat127high-impactattackswereresponsibleforfinancial lossestotaling$2.3billion.StefanosChaliasosetal.,SmartContractand DeFiSecurity:InsightsfromToolEvaluationsandPractitionerSurveys (WorkingPaper),https://www.doc.ic.ac.uk/~livshits/papers/pdf/icse24.pdf. 11AymanAlkhalifahetal.,AMechanismtoDetectandPrevent EthereumBlockchainSmartContractReentrancyAttacks,3 FRONTIERSINCOMPUTERSCIENCE1(2021), ``

`` https://www.frontiersin.org/articles/10.3389/fcomp.2021.598780/f ull. 12ThomasClaburn,SmartContractDevelopersNotReallyFocusedon Security:WhoKnew?Reporter(Apr.26,2022), https://www.theregister.com/2022/04/26/smart_contract_losses/. 13StefanosChaliasosetal.,SmartContractandDeFiSecurity: InsightsfromToolEvaluationsandPractitionerSurveys(Working Paper), https://www.doc.ic.ac.uk/~livshits/papers/pdf/icse24.pdf. ``

Solutions for Smart Contract Vulnerability

`` Adiversesetofproposedsolutionshasemergedtoaddresssmart ``

`` contractvulnerabilitiesissuesandreducetheoverallnumberof attacks.Onepotentialsolutionistheuseofasmartcontract compiler.14Forexample,onesuchcompilerisHCC,which automaticallyinsertssecurityhardeningchecksatthe source-codelevel.15HCCdevelopsacodepropertygraph(CPG)to modelcontrol-flowsanddata-flowsofagivensmartcontract. DuetotheCPGnotation,HCCcanbeappliedtovarioussmart contractplatformsandprogramminglanguages.HCCdevelopers havedemonstrateditefficientlymitigatesreentrancyand integerbugs.16TheyalsoshowhowtointegrateHCCwithinother blockchainplatformssuchasHyperledgerFabric.Their ``

`` evaluationon10kreal-worldcontractsdemonstratesthatHCCis highlypractical.17 ``

`` Alteringthemethodologyofbugclassificationandvulnerability ``

`` analysisprovidesanotherpromisingapproach.Inonestudy, researchersproposetwonewvulnerabilityclasses:distributed ``

14 Jens-Rene Giesen et al., Practical Mitigation of Smart Contract Bugs (Working Paper), <u>https://arxiv.org/pdf/2203.00364.pdf.</u>

15 Id .

16 Id .

17 Id .

`` systemprotocol(DSP)anddistributedsystemresourcemanagement (DRM).18 ``

AI-Driven Solutions

`` TheintegrationofblockchaintechnologyandAIhasimmense promise.19AItechnologyappearsbestsuitedtoeliminatingbugs throughenhancedbugtrackingandbugaudits.Arecentreviewof over100researchpapersrevealedthatintegratingthetwo technologiesresultsincreasesthesecurity,efficiency,and productivityoftheapplications.20 ``

`` Mostbugtrackingisdonemanuallybysoftwareengineers,which impairsbugtriaging.21Toaddressthisproblem,researchers proposePrAIoritize;anautomatedapproachforpredictingsmart ``

18 Wesley Dingman et al., Defects and Vulnerabilities in Smart Contracts, a Classification Using the NIST Bugs Framework, 73 INTERNATIONAL JOURNAL OF NETWORKED & DISTRIBUTED COMPUTING 121 (2019),

`` https://www.atlantis-press.com/journals/ijndc/125913574/view?ref =metastate. ``

19 Rashi Saxena, E. Gayathri & Lalitha Surya Kumari, Semantic Analysis of Blockchain Intelligence with Proposed Agenda for Future Issues, 14 International Journal of System Assurance Engineering & Management 34 (2023),

`` https://link.springer.com/article/10.1007/s13198-023-01862-y. 20Id. 21MajdSoud,GrischaLiebel&MohammadHamdaqa,PrAIoritize: LearningtoPrioritizeSmartContractBugsandVulnerabilities (WorkingPaper),https://arxiv.org/pdf/2308.11082.pdf. ``

`` contractbugprioritiesthatassistsoftwareengineersin ``

`` prioritizinghighlyurgentbugreports.22 ``

`` Enhancedsmartcontractauditingcanbeaccomplishedthrough deeplearningtechniques.Inarecentstudy,researcherstrained threedeepmodelsfordetectingvulnerabilitiesinsmart contract:Optimized-CodeBERT,Optimized-LSTM,and Optimized-CNN.23Experimentalresultsshowthat Optimized-CodeBERTmodelsurpassesothermethods,achievingan f1-scoreof93.53%.24Topreciselyextractvulnerability features,theyacquiredsegmentsofvulnerablecodefunctionsto retaincriticalvulnerabilityfeatures.UsingtheCodeBERT pre-trainingmodelfordatapreprocessing,theauthorscould capturethesyntaxandsemanticsofthecodemoreaccurately. Theauthorsevaluateditsperformanceusingthe SolidiFI-benchmarkdataset,whichconsistsof9369vulnerable contractsinjectedwithvulnerabilitiesfromsevendifferent types. ``

22 Xueyan Tang, Yuying Du, Alan Lai, Ze Zhang & Lingzhi Shi, Deep Learning ‑ based Solution for Smart Contract Vulnerabilities Detection, 13 SCIENTIFIC REPORTS 1 (2023), <u>https://www.nature.com/articles/s41598-023-47219-0.</u>

23 Id .

24 Id.

`` Inanotherstudy,researcherstrainedartificialneuralnetworks (ANN),long-shorttermmemory(LSTM),andgatedrecurrentunit models(GRU)andcomparedtheiraccuracy,precision,recall,and receiveroperatingcharacteristic(ROC)curvevalues.25The networkwastrainedonanopenGoogleBigQuerydatasetwith7000 samples.TheirresultsdemonstratedthattheLSTMmodel outperformsANNandGRU.26Lastly,AItechnologyhasthe ``

`` potentialtogenerallyimprovesmartcontractsecurity.27 ``

Market for Code Reviews

`` Themarketforcodereviewsin2024isdominatedbycentralized marketparticipants.Theexistingcodereviewindustryis ``

`` subjecttosignificantcost,inefficiencies,barrierstoentry, andlackofassurancesforcodereviewjobposters,amongother lackofclientservicesforcodereviews.In2024,thecode reviewindustryisdominatedbyseveralkeyplayerswhocan chargeexorbitantandmonopoly-likeprices.Despitethesehigh prices,thecodereviewprocessissubjecttosignificantflaws. ``

25 Rajesh Gupta et al., Deep Learning-based Malicious Smart Contract Detection Scheme for Internet of Things Environment, 97 COMPUTERS & ELECTRICAL ENGINEERING 1 (2022), <u>https://www.sciencedirect.com/science/article/pii/S0045790621005 19X.</u>

26 Id.

27 Moez Krichen, Strengthening the Security of Smart Contracts through the Power of Artificial Intelligence , 12 Computers 107 (2023), <u>https://www.mdpi.com/2073-431X/12/5/107.</u>

`` Firstandforemost,theselectionprocessforthereviewerisan ``

`` ongoingchallengefortheexistingcodereviewprocessinlegacy codereviewfirms.Themorehierarchicalthecodereview processis,theloweristhequalityofthereviewedcode.It makesintuitivesensethatthemoredevelopersreviewagiven codeset,thehigherthecodequalitymayturnouttobe. However,inthelegacyreviewprocess,thefirstreviewerwithin thehierarchicalstructureofthecodereviewprocessoftengets thehighestpriorityandisoftenmerelyfollowedwithminor upgradesbyfollow-onreviewers.Thecollectiveofreviewersis alsonotincentivizedtofindflawsinthecodetooptimizecode asaworkproductofthecollective.Rather,itisoftenseenas theworkproductoftheinitialreviewerwithminorinputfrom follow-upreviewers.Moreover,themorepeoplereviewthecode withcommentsthataskforclarification,themorelikelyit becomesthatthecodebecomessimplerandclearer,whichinturn typicallyincreasescodequality.However,thatisnotpossible inhierarchicalreviewprocesses.Thehierarchicalapproachto codereviewsundermineslong-termparticipationwithopinions fromtheedgesofthereviewerspectrumbecausethosereviewers eitherhavenoaccessorareinnopositiontohelpreviewthe code.Inotherwords,themorehierarchicalthecodereview ``

`` processandthemorebarrierstoentry,thelowerthequalityof thecode. ``

Single Points of Failure

`` Intheexistinglegacycodereviewprocess,theviewsofthe reviewerandtheintentofthecodeauthorareoftenatodds witheachotherwithoutanycrowdcontrol.Becausethecode reviewermaywishtoimposetheirlogiconthecodeauthor,the codeauthormayberequiredtorewritecodeoverandovereven thoughthecorefunctionalityofthecodeissoundanddangerous issueswerecontrolledfor.Thiscanbehighlytime-consuming andinefficient.Italsocallstheoverallroleofthecode reviewprocessintoquestion.Instead,acodereviewshould focusonthefunctionalityofthecodeandonkeepingmistaken, badlyconstructed,anddangerouscodeout. ``

`` Ifthesingleauthorofacodereviewhasmissedsomethingand thefollow-onreviewerfocusedentirelyonthefirstreviewer's concerns,thecodereviewhasahigherriskoflackofaccuracy. Crowdwisdomisonewaytocorrectpossiblemyopiaandsingle pointsoffailurethroughstandardlegacycodereview. ``

Timing

`` Dependingonthesettingofthecodereview,codereviewsin legacysystemscanlastweeksandmonths.Thiscanbe exacerbatedbymarketconditionsinthedigitalassetmarket. Thesesignificantdelayscanimpactdevelopmentandmayrequire completerewritingofcontractsbecausetheunderlyingprotocol mayhaveupgradedcorelibrariesduringthecodereview. ``

Current Players in the Field

`` Eventhoughmostcodereviewfirmshelpclientswhohopeto decentralizedifferentpartsoftheindustryandcapitalizeon efficienciescreatedbydecentralizinglegacysystems,thecode reviewindustryismainlydominatedbyafewplayers. ``

`` Thecurrentmarketdynamicsdominatedbythetop5auditfirms alsocreatehighbarrierstoentryfornewplayerstoenterinto thecodereviewmarket. ``

`` Giventhesedownsidesintheexistingcodereviewmarket,itis ``

`` abitironicthatoneofthestrongestformsofexploitationand centralizedeconomiesofscalearebeingcreatedinamarket thatis,fromtheoutset,supposedtohelpsupportthe decentralizationofdisparateindustries. ``

Overpricing

`` Asaresultofthecentralizationoftheindustry,mostcode reviewsaresignificantlyoverpriced.Clientsprettymuchpay anypricetogetthestampofapprovalfromoneofthetop5 auditfirms. ``

No Controls

`` Thecentralizedpoweralsounderminesattemptsbyotherindustry playerstocreateinternalorexternalcontrolsonthequality ofcodereviews.Asaresult,thepublichasnoorveryweak controloverthequalityofcodereviewservicesitreceives. jobposterscannotaffordtolookforbetter-pricedcodereviews andareforcedintoconsiderablepricingtoobtainmarket acceptanceoftheirproducts.Inturn,thecentralizationof ``

`` themarketunderminesanyformofdownwardpricepressure. ``

`` Becauseofthepowerofthelimitedplayersovertheoverall marketandtheprocessofthecodereviewanditsoutputs,the qualityofcodereviewisoftensuboptimal. ``

`` Moreover,thereislittleornorecourseforclientsincasesin whichthecodeprovedtobeflawedevenafterfunctionalityand qualityreview. ``

Code Review DAO

`` TheCodeReviewDAO(CRDAO)istacklingmanyoftheissuesthat afflictthemoderncodereviewmarket. ``

`` TheCRDAOprovidesanewcodereviewplatform,whichfacilitates ``

`` adecentralizedcommunity-drivencodereviewprocessthat utilizesabiddingprocessoncodereviewstodrivepricesdown (the“CodeReviewPlatform”).Itprovidesopenaccessforcode reviewsfromanyonewhoqualifies–notjustmembersofthefew codereviewfirms.Atthesametime,itprovidesfull incentivizationforcommunitycodereviewsthroughits decentralizedgovernanceframework. ``

`` Givenitsuniversalaccessandpricediscoverymethodology (throughapublicbiddingprocess),theCodeReviewPlatform createslowbarrierstoentryinthecodereviewmarket.Anyone canjointheCodeReviewPlatformbysubmittinghigh-quality codereviewsthroughtheCodeReviewPlatformportal. ``

`` TheCodeReviewPlatformalsousesacommunitypolicingand auditmethodologyforcodereviews.ThisisenabledbytheCode ``

`` ReviewDAO28(CRDAO)governancemodel.Moreover,theCRDAO governanceandpolicingfunctionsensurelessduplicationof codereviews.CodeReviewPlatformmaintainsstrongincentives forcommunity-drivencodereviewaudits. ``

`` TheCodeReviewPlatformcanfunctionasafirst-roundcode reviewormulti-roundcodereviewwithdifferentcodereview teams.Insummary,jobpostersreceivelow-priced,high-quality codereviewswithcommunityvalidationofreviews. ``

Feedback Loops

`` CodeReviewPlatformprovidesanearlyfeedbacklooponcode reviewsforthedevelopercommunityatamuchlowerpricethan thetraditionalcodereviewandauditmarket. ``

`` TheCRDAOcrowdcontrolsfilteroutidiosyncraticcodereviewer preferences.Codereviewsinlegacycodereviewfirmsareoften highlysubjectivewhichleadstorathersuboptimaloutcomes withoutcrowdcontrols.Nosingledevelopermayagreeonagiven setofcodeanditsintendedfunctionalityandqualityin achievingthecodedobjectives.Thiscanbeattributedto ``

28 DAOs are short for Decentralized Autonomous Organizations. <u>https://en.wikipedia.org/wiki/Decentralized_autonomous_organization#:~:text=Dec entralized%20autonomous%20organizations%20are%20typified,dissemination%20of%20a</u>

<u>%20distributed%20database.</u>

`` differentprogramminglanguageswithdifferentstylesandunique andoftenidiosyncraticpreferences.Instead,theALEmandates thatreviewsaresubjecttocrowdreviewandpolicingvotesby theCRDAOcollective.Accordingly,codereviewersareless likelytoengageinhighlyidiosyncraticreviewsastheywould needtofearslashingofreptokenscoresandlossofstanding inthecommunity. ``

Price Discovery

`` Theexistingcodereviewmarketdoesnotprovidepublicly transparentpricingofcodereviewservices.Assuch,the existingmarketarguablyharmsthepublicforthebenefitofthe fewmarketplayersanditsclients.Intheexistingsystem,the unilateralpricingisnotdisclosedbecauseboththeclientand codereviewermaynotbenefitfrompublicscrutinyofthe prices. ``

`` TheCodeReviewPlatformusesauniqueandfullytransparent pricediscoverymechanismforcodereviews.TheCodeReview Platform’spricediscoverymechanismworksasfollows: ``

- Job poster posts the job on the Code Review Platform portal

- DAO internal - Job price discovery

- Internal DAO bids are collected in a table format next to the job posting terms and show the bids up and down on the terms 1. Time +- 2. Price - + 3. Reputation score range of the bidder (actual reputation of bidder is within the range), 4. Reputation stake of the bidder

- At the end of the internal bidding period - the job poster reviews all the bids and selects the winner and prices are revealed

- If the job poster does not select a winner

- after a grace period, it automatically goes to public bidding and prices are never revealed.

- Bidding is not public during the bidding process and (fully anonymized through reputation score ranges)

- ■ All bids on the post and winner will become public at the end of the selection when the job poster has picked the winner

- Public - Job price discovery

○ If no bids for n days internally in the CRDAO community member pool, the job post will be made public automatically for public bidding - public bidding follows these parameters.

- External bidder posts DoS^29 fee to get onboarded(admin approval)

■ External bids are collected in a table format next to the job posting terms and show the bids up and down on the terms 1. Time +- 2. Price - +

■ At the end of the external bidding period - the job poster reviews all the bids and selects the winner

- If the job poster does not select a winner after a grace period, it automatically ends the bidding process with no winner.

- Bidding is not public during the bidding process and (anonymized if preferred by bidder)

- ■ All bids on the post and winner will become public at the end of the selection when the job poster has picked the winner

29 Denial of Service: https://en.wikipedia.org/wiki/Denial-of-service_attack

● If the job poster accepts a bid (DAO internal or public) - the job poster now has to post a deposit for the value of the agreed code review job into a smart contract.

`` Thispricediscoverymechanismservesakeypublicservice functioninthatitenablesfull-pricetransparencyfor consumersbasedonthevisibilityoftheinternalandexternal bidsonjobposts.Thispricetransparencyisuniqueand unprecedentedinamarket. ``

`` Pricediscoveryisakeypublicservicesfunctionbecause, withoutpublicpricing,consumerscannotrealisticallyselect theserviceproviderthatprovidesthehighestvaluetothejob poster.Thelackoftransparencyenablesinsiderdealstothe detrimentoftheclients,whoareforcedintothepriceagroup offirmsdictates. ``

Standards

`` TheCodeReviewPlatformalsofillsthevoidleftinlegacy reviewswithoutcommonstandards.TheCodeReviewPlatform createsacompendiumofreviewsandthroughitacommonstandard forcodereviewsthatareotherwiselackinginlegacycode ``

`` reviewenvironments.Shouldacollectivereviewcodeundera commonsetofstandards,thestandardshelpguideboththe reviewersandthecollectivetocometoacommonformof expectationsontheappliedfunctionalityandqualityoutcomes forthecode. ``

Speed

`` ThefeedbackprovidedbytheALEforthedevelopercommunity enablesrisk-takingfordevteamswhowishtomovequickly throughtheirgovernanceandupgradeprocess,whichinturn enablesacceleratedgrowthandscalingofexperimentation. ``

Code Testing

`` ThecodereviewsprovidedbytheCodeReviewPlatformprovidea ``

`` firstinstantiationofflathierarchy-drivendecentralized peer-reviewedcodereviews.AlltestingperformedbytheCode ReviewPlatformfollowsthecorestandardsestablishedforthe CRDAOcommunity.Alltestingandstandardsaresubjectto constantreviewandexperimentationandarecontinually, dynamically,andevolutionarilyupdatedinaconstantfeedback ``

`` loopbetweenallconstituents,thatisbetweentheCRDAOmember, ``

`` thejobposters,andpublicbiddersandapplicantsforCRDAO ``

`` communitymemberstatus. ``

Community Audit

`` Theauditstartswiththecommunitydiscussionofthecode review,followedbyaninformalvoteonthecode.Thecommunity thatbidsforthepostedjobsandthenreviewsthesubmittedjob andvotesonitaftertheforumdiscussionisconstitutedbythe evolvingcommunitymembershipoftheCodeReviewPlatform.The informalvoteshowsallCRDAOmembersthecollectivewisdomas appliedtotheworkproductexaminedinthecodereview.Once theentirecommunityknowshoweachmemberfeelsaboutthecode reviewexamined,theCRDAOnowvotesinaformalvoteinwhich thereputationtokens30stakedareatrisk.Thissequenceof votesprovidesjobposterswithsignificantassurancesthatthe codeexaminedandtheCodeReviewPlatformreportonthecode adherestothehigheststandardsofqualityavailable. ``

30 “Reputation Tokens” are non-transferable tokens that cannot be valued and represent the reputation a member has within the community. They simply mirror a scoreboard. Each member of the CRDAO holds such reputation tokens to measure the merit and quality of their input into the community, presenting the member's unique and individual reputation status according to certain criteria established by the CRDAO. At the time of this writing, the criteria for reputation token allocation to CRDAO members revolved around the respective members’ ability to contribute to the CRDAO code reviews, either as a job performer or as part of the community audit.

Code Review Platform Code Review Process

`` ThecodereviewprocessoftheCodeReviewPlatformrevolves aroundCRDAOcommunityengagementwhichminimizesissuesoflack ofcrowdcontrols,lowerstimerequirementsforcodereviews, lowerspricesofcodereviews,increasesdeveloper ``

`` participation,andincreasesoverallfeedback. ``

AI Learning Ecosystem Platform

Scaling the Code Review Platform with Micro Task Work

`` ViatheCodeReviewPlatformasaproofpoint,ALEistapping intothisrisingmicrotaskmarketforAItrainingdataby providinguserswithgamifiedaccesstocodereviews.Key examplesthatsetprecedentforthegamificationincludeAxie InfinityandotherscalingEthereumgamesthatattained worldwideaudiences. ``

Removing Cost of Micro Task Work Duplication

`` Centralizedmicrotaskworkrequiresduplicationtoensure quality.Alltoohumanshortcomingsofmicrotaskworks,suchas limitedattentionspan,irrationality,andinaccuraciesresult ``

`` inverificationrequirementsformicrotaskwork.However, manualverificationofmicrotaskworkissubjecttothesame humanlimitations.Inanattempttoensurequalityofresults andminimizetheimpactofthehumanlimitationsoftheir workers,requestersincentralizedmicrotaskstructuressetup teamsofupto15workerstoperformthesametaskinaneffort toformaconsensus.31Themultiplicationofworkinherentin thisprocesssignificantlyincreasesthecostofmicrotask work.Requiringrequesterstopayproportionally(e.g.upto15 times)forworkperprojectresultsinwaste.Thenecessityof multiplicationofworkalsosubjectsmicrotaskworkerstolower ratesandlackofpaymentincreases. ``

`` Bycontrast,intheALEplatform,thecommunityorganization softwareenablesareadilyavailableindicatorofhowreliablea workerorrequesterisontheALEPlatform.Themicrotask worker’sreputationscoreisameasureoftheworker’shistory ofcompletingmicrotasksontheALEPlatformqualitatively accurate,efficient,andconsistent.Therequester’sreputation scoreisameasureoftherequester’shistoryofinteracting withmicrotaskworkersontheALEnetwork.TheALEreputation ``

31 Neeraj Kumar, Effective Use of Amazon Mechanical Turk (MTurk); Tips and techniques for better usage of Amazon Mechanical Turk for researchers, NEERAJ KUMAR (May 2013, updated May 8, 2014) , http://neerajkumar.org/writings/mturk/;

Rory O’Reilly, How the Gems Protocol Reduces Consensus by Redundancy , GEMS (Nov. 27, 2017),

https://blog.gems.org/how-the-gems-protocol-reduces-consensus-by-redundancyb151de80ecb8.

`` scoreisformedandlinkedtotherespectivenetwork participant’swalletaddress. ``

`` Marketfactorsbalancetheequilibriumofsupplyanddemandof microworkontheALEPlatformbasedontheworkersand requestersreputationscores.Ifrequestershavealower reputationscore,workersbecomelesslikelytoaccept requesters’offers.Inturn,lowreputationscoresformicro taskworkersresultinalowerlikelihoodofretentionformicro taskworkontheALEPlatform.Requesterscanselectworkers basedontheirreputationscore,givingworkersanincentiveto keepthereputationscoreshighbyperformingmicrotaskswith highaccuracyandefficiency.Thereputationscoremechanismand thebuildingofreputationontheALEPlatformallowsworkersto graduatetotheprivilegeofbeingaverifier.Thereputation scoremechanismhelpsdiscernmaliciousactorsandsimple mistakes.Italsoprotectsworkersandverifiersfromfraudulent requestersandsuboptimallydesignedrequests. ``

`` Insummary,themarketfactorsandmarketdynamicsrelatedto ALEPlatformreputationscoresenablealoweringofthecostof duplicationascomparedtocentralizedmicrotaskwork.Ifa reliableandhighreputationscoreworkercompletesthetasks, theduplicationmaybebroughtfrom15to5orlessinthe ``

`` decentralizedcodereviewsetup.Thisenablesunprecedented ``

`` scalingofmicrotaskwork. ``

Transaction Cost Minimization of Micro Task Work

`` TheALEPlatformremovestransactioncostsassociatedwithmicro taskwork.Unlikecentralizedmechanicalturkplatformsthat requireanexistingbankingrelationshiptoreceiveaccount transfersforotherwiseunbankedmicrotaskworkers,theALE Platformoperatesentirelythroughcryptotransactions. ``

Gamification of Micro Task Work

`` Thegamificationofmicrotaskworkcanhaveseveraleffectson theprocessofproducinghigh-quality,diversedatasetsforAI learningviatheALEPlatform. ``

`` 1.IncreasedEngagement:GamificationtechniquesoftheALE Platformincludeapointsystem,levels,leaderboards,and rewards,allofwhichmaybeutilizedexclusivelywithinthe platform,thatareallocatedviareputationscoresinthe community.ThissystemofreputationscoringintheALEPlatform ``

`` enhancestheengagementofcrowdworkersintheALEPlatform.By introducingthereputationscoresasacompetitiveandrewarding element,theALEPlatoform’sgamificationmotivatesworkersto ``

`` activelyparticipateandcompletetasksmoreefficientlyand moreresponsively.Thisincreasedengagementcanleadto higher-qualityoutputsasworkersaremoreinvestedinthe process.Workersgainreputationscoresandarerewardedwitha partofanyincomingcompensation(stablecoinsorothermajor cryptocurrenciespaidbyjobposters)proratatotheir reputationscores.Thisincentivizesthemtocareaboutthe reputationscoreandengagewithmorerigorandduediligencein themicrotaskwork.Iftheydon’tengagewiththecare required,theysacrificetheirspotintherankingsof reputationscoreswhichaffectstheirparticipationinthejob feedistribution,whichispaidoutproratatotherespective reputationscores. ``

`` 2.QualityControl:GamificationintheALEPlatformimproves dataqualitythroughtheconsensus-basedvotingonworkproducts ofindividuals.WorkersintheALEPlatformreviewandrateeach other'scontributions,earningreputationpointsforaccurate andconsistentwork.Thisapproachhelpsidentifyandresolve discrepancies,buildconsensus,andensuresbetterquality controlinthedatasetcreationprocess. ``

`` 3.SkillDevelopment:GamificationintheALEPlatformis focusedonexpertisebuildingandthusenablesskilldevelopment amongcrowdworkers.Byprovidingchallengesofworksetsin levelsthatprogressivelyincreaseindifficulty,workerscan enhancetheirannotation,validation,andcleaningskillsover time.Allskilldevelopmentisdirectlytraceablethroughthe reputationscoreofeachworker.Thisskilldevelopmentcan resultinimprovedaccuracyandefficiencyindatasetcreation. ``

`` 4.CrowdworkerRetention:GamificationelementsontheALE Platformlikebadges,achievementawards,andvirtualcurrency paymentsthatareallrelatedtothereputationscoreofeach workercanenhancecrowdworkerretention.Byrecognizingand rewardingworkersontheALEPlatformfortheircontributions, gamificationfostersasenseofaccomplishmentandencourages workerstocontinueparticipatinginmicrotaskwork.This retentioniscrucialformaintainingaconsistentpoolof experiencedandreliableworkersfordatasetcreation.Attrition ratescanfallthroughthegamificationdesigninherentinthe ALEPlatform. ``

`` 5.ScalabilityandSpeed:GamificationontheCodeReview Platformcanhelpexpeditethedatasetcreationprocessby encouragingworkerstocompletetasksmorequickly.Thisisdone inanefforttomaintaintheirreputationscores.Byintroducing time-basedchallengesinprovidingtheworkandbyoffering bonusesfortimelycompletion,gamificationCodeReview Platformcanincreasethespeedatwhichhigh-qualitydatasets aregenerated,enablingscalabilityforlarge-scaleAIprojects. ``

`` TheALEPlatformemphasizesthedesignandimplementationof itsprovendecentralizedgovernancelogicincombinationwith thegamificationlogicinherentinitsreputationgovernance. Throughthecombinationofdecentralizedgovernanceand gamificationofmicrotaskwork,ALEPlatformensuresthat gamificationtechniquesdonotcompromisethequalityand accuracyofthedataset.Balancingthegamificationelements withappropriatequalitycontrolmeasuresviadecentralized governanceisessentialtomaintaintheintegrityofthedataset creationprocess. ``