Strong encryption algorithms and reliable anonymity routing have made cybercrime investigation more challenging. Hence, one option for law enforcement agencies (LEAs) is to search through unencrypted content on the Internet or anonymous communication networks (ACNs). The capability of automatically harvesting web content from web servers enables LEAs to collect and preserve data prone to serve as potential leads, clues, or evidence in an investigation. Although scientific studies have explored the field of web crawling soon after the inception of the web, few research studies have thoroughly scrutinised web crawling on the "dark web" or via ACNs such as I2P, IPFS, Freenet, and Tor. The current paper presents a systematic literature review (SLR) that examines the prevalence and characteristics of dark web crawlers. From a selection of 58 peer-reviewed articles mentioning crawling and the dark web, 34 remained after excluding irrelevant articles. The literature review showed that most dark web crawlers were programmed in Python, using either Selenium or Scrapy as the web scraping library. The knowledge gathered from the systematic literature review was used to develop a Tor-based web crawling model into an already existing software toolset customised for ACN-based investigations. Finally, the performance of the model was examined through a set of experiments. The results indicate that the developed crawler was successful in scraping web content from both clear and dark web pages, and scraping dark marketplaces on the Tor network. The scientific contribution of this paper entails novel knowledge concerning ACN-based web crawlers. Furthermore, it presents a model for crawling and scraping clear and dark websites for the purpose of digital investigations. The conclusions include practical implications of dark web content retrieval and archival, such as investigation clues and evidence, and the related future research topics. 

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

The increasing complexity and connectivity of automotive systems have raised concerns about their vulnerability to security breaches. As a result, the integration of formal methods and validation techniques has become crucial in ensuring the security of automotive systems. This survey research paper aims to provide a comprehensive overview of the current state-of-the-art formal methods and validation techniques employed in the automotive industry for system security. The paper begins by discussing the challenges associated with automotive system security and the potential consequences of security breaches. Then, it explores various formal methods, such as model checking, theorem proving, and abstract interpretation, which have been widely used to analyze and verify the security properties of automotive systems. Additionally, the survey highlights the validation techniques employed to ensure the effectiveness of security measures, including penetration testing, fault injection, and fuzz testing. Furthermore, the paper examines the integration of formal methods and validation techniques within the automotive development lifecycle, including requirements engineering, design, implementation, and testing phases. It discusses the benefits and limitations of these approaches, considering factors such as scalability, efficiency, and applicability to real-world automotive systems. Through an extensive review of relevant literature and case studies, this survey provides insights into the current research trends, challenges, and open research questions in the field of formal methods and validation techniques for automotive system security. The findings of this survey can serve as a valuable resource for researchers, practitioners, and policymakers involved in the design, development, and evaluation of secure automotive systems.

Formal Methods and Validation Techniques for Ensuring Automotive Systems Security

Penggunaan teknologi dalam berbagai bidang meningkatkan mobilitas, salah satunya dengan pembuatan website untuk berbagi dan mengelola informasi. Keamanan sistem informasi yang tidak baik dapat mengganggu infrastruktur suatu organisasi atau perusahaan. Masalah kerentanan atau gangguan keamanan sistem banyak terjadi di internet. Masalah tersebut dapat berjenis serangan Malware, Eksploitasi dan Injeksi database. Masalah tersebut dapat diminimalisir dengan menerapkan pengamanan web dari gangguan atau serangan hacker dengan cara penetration testing (Pentest) yaitu pengujian yang dilakukan terhadap web secara legal dengan aktifitas menyerupai hacker. Untuk mendeteksi keamanan web dibutuhkan sebuah analisis terhadap kerentanan sebuah web yang sesuai dengan standarisasi kemanan Open Web Application Security Project (OWASP) dengan menggunakan tools security. Analisis kerentanan aplikasi berbasis web dengan metode OWASP dengan menggunakan tools security mampu mengetahui tingkat keamanan suatu aplikasi, berdasarkan hasil pengujian yang telah dilakukan dimana hasil dari penelitian memberikan beberapa saran atau rekomendasi tentang kerentanan situs web, yang dapat digunakan oleh tim developer situs web untuk meningkatkan keamanan situs web tersebut.

Analisis Metode Open Web Application Security Project (OWASP) Menggunakan Penetration Testing pada Keamanan Website Absensi

An automated approach to Web Offensive Security

A Method for Automatic Penetration Testing and Mitigation: A Red Hat Approach

In the domain of web security, websites strive to prevent themselves from data gathering performed by automatic programs called bots. In that way, crawler traps are an efficient brake against this kind of programs. By creating similar pages or random content dynamically, crawler traps give fake information to the bot and resulting by wasting time and resources. Nowadays, there is no available bots able to detect the presence of a crawler trap. Our aim was to find a generic solution to escape any type of crawler trap. Since the random generation is potentially endless, the only way to perform crawler trap detection is on the fly. Using machine learning, it is possible to compute the comparison between datasets of webpages extracted from regular websites from those generated by crawler traps. Since machine learning requires to use distances, we designed our system using information theory. We considered widely used distances compared to a new one designed to take into account heterogeneous data. Indeed, two pages does not have necessary the same words and it is operationally impossible to know all possible words by advance. To solve our problematic, our new distance compares two webpages and the results showed that our distance is more accurate than other tested distances. By extension, we can say that our distance has a much larger potential range than just crawler traps detection. This opens many new possibilities in the scope of data classification and data mining.

https://link.springer.com/content/pdf/10.1007/s11416-021-00380-4.pdf

Detection of crawler traps: formalization and implementation—defeating protection on internet and on the TOR network

Popular Tasks used by Researchers

How can I help with your research?

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

We encountered an error while loading the panel. Please try refreshing the page.

Something went wrong

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

Anyone with the link can view this chat. You can switch it back to private anytime.

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Eric Filiol

Author Tools

Chat about Author

Papers

A Method for Automatic Penetration Testing and Mitigation: A Red Hat Approach

Detection of crawler traps: formalization and implementation—defeating protection on internet and on the TOR network