Data Collection and Clearing: Sources, Formats, and Scrubbing

The first two phases of the OSEMN framework — collect (obtain) and clean (scrub) — are the foundation of every successful data project. If you start with incorrect or incomplete data, all subsequent analysis leads to incorrect results. In this article, we explain where to find the data, what formats it comes in, and how to clean it.

This post what is data analytics a continuation of its guide.

Where to find the dataset? 4 main sources

When you encounter a data question, you can reach out to different sources. They can be divided into four groups:

1. Free, open-source

Government websites and open repositories provide socio-economic information (population demographics, household income, education) for free. Examples: Eurostat (official EU statistics), OECD (Organisation for Economic Co-operation and Development), World Bank, UN and US Census Bureau. These sources are reliable and up to date.

2. Company-specific data

Your own aggregated and non-public dataset of the company you work for. For example, a system that records all sales, or Google Analytics, which is used to track the site. The phone company customers' information about their call and data usage also applies.

3. Subscribed bases

Companies subscribe to data for their fields. Example: Nielsen (people's media usage — which TV channel they view, which websites they visit) and the Bloomberg terminal (paid financial information for financial companies). The Nielsen dataset, for example, allows the TV channel to show advertisers the number of sports fans.

4. Self Collected Data

If the necessary data isn't available, you'll need to create it yourself: a survey (questionnaire), an interview, or observing user behavior. SurveyMonkey and Google Forms are easy, ready-made template tools for this.

What formats does the data come in?

Data isn't just a number-it can also be text, image, video, or voice. Three main formats:

Numeric (digital) data

Quantitative data — digitized and measurable: age, income, sales figures, stock prices. Often saved as a .CSV file in most spreadsheet (tabular) format — fields of rows and columns. This structure allows you to quickly process large volumes of data with sorting and filtering.

Text (text) data

Structured data — letters and words. Sources: social media posts, emails, tweets, blog posts, customer reviews. Often, natural language processing (NLP) is analyzed-for example, by sentiment analysis to determine what customers feel about the product.

Visual data

Pictures and videos. Examples: product images for production quality control, road images captured by self-driving cars' cameras (to teach machine learning algorithms), and medical imaging. Depending on its purpose, sometimes visual data is more useful than digital ones.

Structured and unstructured data

An important part of formats is between structured and unstructured data. Structured data fits neatly into rows and columns — the table. Digital data is usually like this, and it's easy to work with: sort, filter, total. Structured data, however, does not fit into this framework: text, image, video, voice. It is more difficult to work with and often requires special techniques (NLP, image recognition).

In many real projects, both types come together. For example, in an e-commerce store, sales numbers are structured, while customer reviews are unstructured. A good analyst combines the two: the numbers show "what" happened, and the reviews show "why". Knowing what structure the data has determines how to process it.

Scrubbing: cleaning the dataset

The second step — scrubbing (cleaning) — turns the raw, “dirty” dataset into a clean dataset. Dirty data contains duplicate records, inconsistent format, missing values, or incorrect information. And clean data consists of only unique records, has a consistent structure, has no missing value, and is valid. The cleaning process is broken down into four main tasks:

1. Remove duplicates — remove duplicates of the same record.
2. Formatting records (format records) — creating a consistent structure (for example, putting all sizes into the same system).
3. Resolving missing values (handle missing values) — deciding how to deal with empty spaces.
4. Check for incorrect values — detecting incorrect and unrealistic information.

Why is it so important? Because errors in the dataset can disrupt research and analysis, bring false conclusions, and even prevent the model from working at all. In Anna's boutique example, some sizes were «small/medium/large», some were American figure sizes, some were Italian sizes — she turned them all into a single category. Taking the time to clean up is worth protecting against later issues.

Deleting repeats

The results are distorted when the same note appears multiple times in the dataset. For example, if the same customer is counted twice, it shows more than the number of customers. Identifying and removing repetitions is the first step to a clean start.

Formatting notes

The data must be in a consistent format. Dates should be set to the same format (e.g. GG/AA/YYYY), texts should be set to the same uppercase/lowercase standard, measurements should be made to the same system. Without formatting, the program can treat the same information as different — «Baku» and «baku» can be seen as two separate values.

Solving missing values

Some fields may be empty. How should I deal with these? Options: remove that line, fill the gap with a medium or median, or place a special “unknown” value. The choice depends on the type of dataset and how many spaces there are.

Checking for false values

Some values may not be realistic: 250 years old, negative price, or future date sale. These anomalies need to be detected and corrected, otherwise they confuse the analysis.

Sampled and first/third-party data

Data toplayanda iki əlavə anlayışı bilmək lazımdır. Birincisi — sampled (seçmə) data. Bəzən bütün əhalini deyil, təmsil edici bir nümunəni araşdırırsan. Bu, sürəti artırır, amma nümunə kifayət qədər böyük və təmsil edici olmasa, nəticə yanıltıcı ola bilər. İkincisi — datanın mənbəyə görə növü. First-party data birbaşa öz müştərilərindən toplanır (ən dəqiq və sənə məxsus). Third-party data isə kənar tərəf tərəfindən toplanıb satılır — daha geniş, amma az dəqiq və çox vaxt məxfilik baxımından həssasdır. Analiz üçün datanı seçəndə bu fərqləri nəzərə almaq keyfiyyətli nəticə üçün əsasdır.

In practice, when you start a project, think ahead about how to retrieve the dataset: are there free sources available, what data does the company have access to, how will you collect it if you don't have the necessary data? This makes all the next steps in planning easier.

Reliability of data sources

When collecting data, it is important to assess its reliability: is the source trusted, is the data up to date, how was it collected? Government and international organization sources are usually valid. When receiving data from an outside source, however, you need to be able to see it firsthand and know how it is collected-otherwise, it may be low quality or risky for privacy.

Frequently asked questions (FAQs)

What are the best free data sources?

Eurostat, OECD, World Bank, UN and US Census Bureau — reliable and free sources for socio-economic data.

Is scrubbing the same thing as cleaning?

Yes. Scrubbing and data cleaning mean the same process — turning a dirty dataset into a clean dataset.

What is CSV?

Comma-Separated Values is a simple table file format that stores the digital data in rows and columns.

Why is data cleaning so important?

Because dirty data analysis can disrupt, bring false conclusions, and even prevent the model from working. Taking the time to clean prevents further major issues.

How is text data analyzed?

Often identify the relationship (positive/negative) between natural language processing (NLP) — for example, sentiment analysis and customer feedback.

What is the difference between structured and unstructured data?

Structured data fits into a table (row/column) — like numbers. Structured data, however, doesn't fit — like text, images, videos; special techniques are needed for processing.

When is sampled data used?

A representative example is when the entire dataset is too long or expensive to process. The sample should be large enough and representative, otherwise the result may be misleading.

How to deal with missing values

Options: remove that string, fill the gap with a medium/median, or leave a value of “unknown”. Selection depends on dataset type and cavity size.

What are the 4 main sources of data collection?

Free open source (Eurostat, OECD), company-specific data, subscribed bases (Nielsen, Bloomberg), and self-published data (survey, interview).

The result: good analysis starts with clean data

The steps of gathering and cleaning may not look so appealing — neither are they «smart» enough to build a model, nor are they «beautiful» enough to visualize. But the foundation of all reliable analysis is right here. If you collect data from a faulty or biased source, or if you don't clean it up properly, the whole next thing — no matter how perfect — will come to the wrong conclusion. That's why experienced analysts pay close attention to these steps.

Practical rule: plan where to find the dataset in each project, check the reliability of the source, understand the format, and allow enough time to clean up. This habit will save you a lot of trouble in the long run and give you confidence in the results of their analysis. Remember: without pure data, the most perfect model yields the wrong result.

Dirty data vs. clean data: a practical example

Let's be specific about the difference. In a dirty customer dataset, the same customer may be mentioned twice (again), dates may be in different formats (01/02/2026 and 2026-02-01), some fields may be blank (missing value), and a customer's age may be unrealistic like «250» (incorrect value). In the clean version, every customer comes once, dates are in the same format, without blank space, and with real values. These four problems fit exactly into the four tasks of scrubbing.

This is where the majority of data analytics takes place. For some assessments, analysts spend most of their time cleaning the dataset because without clean data, the next steps have no meaning. This may seem “boring,” but it's the foundation of all valid analysis.

Practical workflow: collection and cleaning

What do these two steps look like in a project? Let's say you're investigating the traffic drop of your site. First, you collect the dataset: extract the site dataset from Google Analytics, perhaps adding customer feedback with a survey. Then you clean up: you delete repeat sessions, put dates into a single format, solve empty boxes, and check for unrealistic values (like 10 hours remaining on the site). Only then are you ready to explore the dataset.

These two steps may seem boring, but they are the foundation of the entire analysis. Experienced analysts know that the majority of their time is spent here — and it's not a wasted time. If you start with clean, reliable data, the subsequent steps of research and modeling will be both easy and straightforward.

Next step: explore a clean dataset and learn how to build a model.

👉 Next Post dataset investigation and modeling