The ‘Deep Web’ is a term used to describe digital objects and data that are not accessible over the internet via search engines((INTERPOL. The Dark Net. (2018). Available at: https://www.interpol.int/Crime- areas/Cybercrime/The-threats/The-Darknet.)). It comprises protected and proprietary data (such as bank accounts, email and research data) as well as intranets. About 96% of all online data is in the Deep Web (DW), including most of the world’s Instant Messaging (IM) traffic.
A smaller fraction is known as the ‘Dark Net’, including hidden sites developed and used by people that require anonymity. The Dark Net community uses specialized platforms that obfuscate the IP address and their identity; the most popular is Tor (‘The Onion Routing’), based on software developed by the US Naval Research Laboratory in the mid-1990s((Barlow, M. & Fell, G. Patrolling the Dark Net: What You Don’t Know Will Hurt You. (O’Reilly Media, Inc., 2016).)) and released in 2002. Tor is a free software that routes internet traffic through an open voluntary network of about 10,000 nodes to hidden services that use the onion address.
While the total traffic flow is unknown (users are, by definition, anonymous), estimates are indicative: The Tor project claims its traffic accounts for a very small (an estimated half a percent) of all global internet traffic((Cole, J. Dark Web 101. (Air Force Air Command and Staff College Maxwell AFB United States, Air Force Air Command and Staff College Maxwell AFB United States, 2016).)). A criminal fraction, however small, is still a dangerous fraction. About 50% of the activity in the Dark Net is considered illegal((Cole, J. Dark Web 101. (Air Force Air Command and Staff College Maxwell AFB United States, Air Force Air Command and Staff College Maxwell AFB United States, 2016).)) in most jurisdictions.
The Dark Net has become particularly known for the global illegal drugs trade (see figure 1), but it is also a significant source of leaked corporate data, access credentials and cyber weapons (see figure 2). In February 2016 an analysis of 29,532 .onion sites found that 43% were associated with leaked data, financial fraud and hacking.
Within the Deep Web, the inaccessible and anonymised Dark Net is a hinterland for cyber crime. Cyber criminals succeed because attribution is very difficult((Rid, T. & Buchanan, B. Attributing Cyber Attacks. J. Strateg. Stud. 38, 4–37 (2015). )) and prosecutions too few: the timeline between crime and conviction may be many months or years. While the definition of ‘cyber crime’ depends very much on the perspective, there are crimes which cross the line: Activities that cause physical harm and losses in the real world; and at the worst, abuse of (universally accepted) human rights. Even several of the illicit Dark Net marketplaces explicitly exclude these activities((Sui, D., Caverlee, J. & Rudesill, D. The Deep Web and the Darknet: A look inside the internet’s massive black box. (Woodrow Wilson International Center for Scholars, 2015).)). The Dark Net is, either by design or accident, part of a global ecosystem that sustains crime.
Figure 1: The global digital trade in illegal drugs. Image source – The Economist((The Economist. Shedding light on the dark web – Buying drugs online. (2016).))
Figure 2: A February 2016 analysis by threat intelligence companies, categorising 29,532 .onion hidden services. Source: Intelliag / Darksum / Securityaffairs.co((Paganini, P. Mapping the Dark Web searching for illegal content. Security Affairs (2016). Available at: https://securityaffairs.co/wordpress/46202/deep-web/dark-web-mapping.html.)).
As at March 2018, Darkwebnews.com lists 73 ‘markets’. The demise of the Silk Road in 2013 with the prosecution of its American creator, the takedown of Silk Road 2 shortly thereafter and the removal of two other major markets – AlphaBay and Hansa – in July 2017((Europol. Massive blow to criminal Dark Web activities after globally coordinated operation. Europol Available at: https://www.europol.europa.eu/newsroom/news/massive-blow-to-criminal-dark- web-activities-after-globally-coordinated-operation. )) (see figure 3, below), suggest that law enforcement agencies are increasingly successful in finding methods for combatting cyber crime – even if detections and prosecutions take months or years. Meanwhile, as an innovation marketplace for criminals, it is very hard to estimate if the Dark Net is diminishing in scale.
[table id=6 /]
Taken together, the Deep and Dark Net (DDW) comprise a digital world that relies on – and enables – anonymity, virtual currencies and trade in data (legal or otherwise). Like the criminal cosystems it serves((Duijn, P. A. C. & Sloot, P. M. A. From data to disruption. Digit. Investig. 15, 39–45 (2015). )), the Dark Net is continually evolving, challenging the ability of law enforcement teams to keep up((Ellis, C. Serious and Organised Crime in the Digital Age. RUSI (2015). Available at:
https://rusi.org/commentary/serious-and-organised-crime-digital-age.)). Advanced encryption is now a mainstream technology, enabling criminal gangs to operate successful business models. Digital crime is a fixed feature of the crime landscape((Hannigan, R. Criminal cyber gangs are disrupters in the digital economy. Financial Times (2017). )). The Dark Net can also be a threat multiplier: Europol are targeting the DDW and Internet of Things (IoT) which in combination present a serious threat vector for cyber attack.
Given the apparently lawless hinterland created by the Dark Net, it is reasonable to question why it is still allowed to exist. There are three possible explanations for this:
- criminal networks are just too complex to track in real-time;
- those in favour argue that the Dark Net facilitates privacy and anonymity, particularly in places where citizens do not enjoy freedom of speech;
- of all the international spaces, cyberspace is probably the most difficult to police: laws that work well in one sovereign state do not easily translate to others, and – while some borders in cyberspace do exist((Finklea, K. Dark Web. US Congressional Research Service (2015).)), e.g. sites belonging to state entities, servers hosted in a single jurisdiction – cyberspace is beyond any conventional definition of human geography.
The easier target may therefore be to “focus systematically on bad actors and bad patterns”((Sui, D., Caverlee, J. & Rudesill, D. The Deep Web and the Darknet: A look inside the internet’s massive black box. (Woodrow Wilson International Center for Scholars, 2015).)). It is perhaps inevitable that technology once designed to protect is also the enabler of crimes that generate misery and human rights abuses.
The Deep Web and Dark Net: A new frontier for Competitive Intelligence?
Figure 3: Shutting down the markets. Image source: Europol((Europol. Massive blow to criminal Dark Web activities after globally coordinated operation. Europol Available at: https://www.europol.europa.eu/newsroom/news/massive-blow-to-criminal-dark- web-activities-after-globally-coordinated-operation.))
Since the time that people first sought advantage in new terrain, the tools and practice of Competitive Intelligence have informed their strategy. As a discrete business activity, Competitive Intelligence (CI) was defined in the post-war period during a time when knowledge became a valuable asset that could make and break business ventures. CI was formalised in the early 1990s((Wright, S. Competitive intelligence, analysis and strategy: creating organisational agility. (Routledge, 2015).)) along with other knowledge management practices that are today experiencing their next phase of evolution in the era of advanced search algorithms and Artificial Intelligence. At core, CI is about collecting information that will guide a person or an organisation in understanding their competition and/or experimenting with a new business approach.
The advent of data sharing via the internet opened a new dimension to CI: Even before search engines indexed the global digital surface, it became possible to gain insight at-a- distance. Deep Web search solutions have been around for at least 15 years((Bergman, M. K. White Paper: The Deep Web: Surfacing Hidden Value. J. Electron. Publ. 7, (2001).
)) and are well- known in the academic research community: Some of the solutions have found ready utility in the corporate sector((Deep Web Technologies, Inc. Deep Web Tech. (2016). Available at: https://www.deepwebtech.com/.)). In the past five years, companies have begun to recognise the value of the DW for CI. Germany’s Institute for Competitive Intelligence offers training in CI for the deep web((ive Intelligence. ICI-4 Workshop: Advanced Web Intelligence (OSINT – Open Source Intelligence). Institute for Competitive Intelligence (2018). Available at: https://www.institute-for- competitive-intelligence.com/ici-workshops/ici-4-advanced-web-intelligence-osint-open-source- intelligence.)).
Figure 4: The Dark Net is a subset of the Deep Web: Both contain digital objects and communities that are useful to professionals in competitive intelligence and business risk management.
In the digital age of competitive intelligence, the Dark Net is also a data source – for those equipped and trained to navigate it (see figure 4). The DDW is the ‘next level’: it requires highly specialised skills, domain knowledge and ready access to legal counsel. As in cybersecurity, commercial DARKINT (dark web intelligence) is a growing and profitable field.
In March 2016, a Motherboard journalist observed that two groups of people make money from the dark web: Criminals and tracking companies who consult for law enforcement. Cyber threat intelligence companies can make about $150,000 (€122,000) per client, per year((Cox, J. The Booming, and Opaque, Business of Dark Web Monitoring. Motherboard (2016). Available at: https://motherboard.vice.com/en_us/article/vv7b4m/the-booming-and-opaque-business- of-dark-web-monitoring. )). A third group can now be added: business risk intelligence analysts.
Companies with budget and risk appetite venture into the Dark Net for:
- Threat intelligence((Recorded Future. Mining Threat Intelligence from the Dark Web. Recorded Future (2017). Available at: https://www.recordedfuture.com/dark-web-threat-intelligence/.)), analysed (with caution) as part of cybersecurity practice((Polancich, J. The Dark Web: An Untapped Source For Threat Intelligence. Dark Reading (2015). Available at: https://www.darkreading.com/analytics/the-dark-web-an-untapped-source-for- threat-intelligence-/a/d-id/1320983. ));
- Breach response – in cases where data has been lost and traded;
- Counter-corporate espionage (to identify indicators of attack timelines and proprietary data searches) and business risk intelligence.
)) (Germany and Switzerland), corma (Germany)((Weber, J. Dark Net Investigations. corma GmbH Available at: https://corma.de/en/investigations/dark-net-investigations/. ))
DDW investigations and analysis is not for the uninitiated. Operations must be precise in intent and in scope to deliver business value((Deloitte. Dark analytics. (2017).)). Whether the objective is to acquire market insight, identify an insider threat or defend a companyies reputation, DDW analysis is generally not a task for the in-house information security team – however good they may be – because navigating, analysing and understanding what the DDW contains requires specialist skills that have little to do with technical expertise((Lefkowitz, J. Why ‘DIY Deep & Dark Web Intelligence’ is a Bad Idea. Security Week (2017). Available at: https://www.securityweek.com/why-diy-deep-dark-web-intelligence-bad-idea.)). Infiltrating a chat forum, for example, requires knowledge of the communication style, vernacular, online vocabulary and group dynamic; and for materials that are not in English, advanced language skills to understand not only the context and the vernacular, but also its intent. In this context, AI and complex systems analysis have an important role in the new generation of network profiling techniques((Duijn, P. A. C. & Sloot, P. M. A. From data to disruption. Digit. Investig. 15, 39–45 (2015). )).
Figure 5: Building a digital forensics team requires specialised soft skills; many of the objectives may run counter to corporate cybersecurity strategy and knowledge management practice.
Even supposing that, say, a trove of corporate data is identified for sale in the Dark Net, the seller will rarely place the entire haul in the marketplace((Ely, A. Tracking Stolen Data on the Dark Web. IANS (n.d.). Available at: https://www.iansresearch.com/insights/reports/tracking-stolen-data-on-the-dark-web.)): A subset is offered for appraisal. The intelligence analyst then has to decide if the asset is genuine and what the value to the company would be of retrieving it and/or pursuing charges against the hawker.
Getting results from research in the Dark Web is not easy and the costs can be hard to justify. It requires strict information management discipline, a working digital forensics strategy (which may not always mesh easily with the organisation’s cybersecurity strategy) and specialised skills in at least three domains: Languages and the use of vernacular in online conversations; ability to establish trust in a digital community; analytical skills to discern genuine insight from noise((Lefkowitz, J. Why ‘DIY Deep & Dark Web Intelligence’ is a Bad Idea. Security Week (2017). Available at: https://www.securityweek.com/why-diy-deep-dark-web-intelligence-bad-idea.)). The program also requires budget for on-demand legal counsel((Ely, A. Tracking Stolen Data on the Dark Web. IANS (n.d.). Available at: https://www.iansresearch.com/insights/reports/tracking-stolen-data-on-the-dark-web.)) and continuous learning (see figure 5). The aim of any hunting expedition should be to produce objective evidence, that ideally will stand up to scrutiny in a court of law, while remaining accountable at every step.
This presents considerable challenges for most threat intelligence operations, because establishing ground truth of information((Cox, J. The Booming, and Opaque, Business of Dark Web Monitoring. Motherboard (2016). Available at: https://motherboard.vice.com/en_us/article/vv7b4m/the-booming-and-opaque-business- of-dark-web-monitoring.)) obtained through the internet is difficult, expensive and constrained by laws in most jurisdictions.
