Technically Impossible

Lets look at the weak link in your statement. Anything "Technically Impossible" basically means we haven't figured out how yet.

Rational solution for blocking online ads - Hosts, DNS proxy, and Local DNS

Abstract

Recent online advertisement is just like SPAM.

  • heavily use of grotesquery photos
  • improper comics in online book store ads.
  • poor parental control

Online ads somehow brings unpleasant contents as if they have pipe through parental control.

I set the Content Restrictions —> Apps to 4+ years old but it still showed inappropriate ads for games that is for older ages.

How to stop inappropriate ads - Apple Community

Considering lack of platformer's responsibility, I feel the necessity of self-defense solution for blocking unwanted online ads.

Online ads have one aspect as necessary evil to provide free contents with sponsored. sponsor. In essence, contents of advertisement is obnoxious, but ad itself is not. Although blocking only images is one solution to block minimized targets, this solution doesn't work. Sometimes advertisement containing images is generated as a widget, and even if trying to block only images, it ends up block entire advertisement.

After a few days of study, I came to the conclusion that a combination of hosts file, DNS proxy, and local DNS might be a realistic approach. This "realistic" indicates that none of them are perfect, and each has its own advantages and disadvantages.

First of all, a conclusion

It is common to utilize Hosts file to block online advertisement, but it is difficult to edit it on Android, and it is not realistic to do it on iPhone and iPad. In accordance with the assumptions and policies described later, this topic concludes that followings are rational solution.

best local DNS PC almost everythng under control
easy management of block domain
2nd best local DNS proxy PC dependency with services as AdGuard
easy management of block domain
realistic DNS proxy + Hosts PC
Android
difficulty of Hosts editing on Android
common Hosts PC
Android
diffifulty of subdomain management
difficulty of Hosts editing on Android
common DNS proxy PC
Android
iPhone, iPad
complete service dependencies
sole solution for Apple product

Assumptions, policies

In this post, I aim for a solution that is as controllable by users as possible. There are many apps and DNS services to block online ads. Although they provide block capability, they also bring following risks as well.

unexpected blocking subdomain you want to block might not be blocked.
subdomain you DON'T want to block might be blocked.
inappropriate resolver inappropriate IP address might be respond.
selling off*1 don't know how long it will be available.
it might be modified.

I want to avoid situations where the user is unaware that something that should be displayed is not, or that the site does not work properly due to unintended blocks. I decided to minimize the use of these apps and services, and use only standard technologies whenever possible to create a solution. The policy is as follows.

  • Identify blocked domains to the best of your ability.
  • Minimize the operation and management of blocked domains.
  • Operate as usual in domains other than the blocked domain.

Hosts file

Applicable PC
Android
Pros Can be stored in local environment
Cons Vast number of subdomains for registration candidates
Risk of distributed Hosts definitions

Since the Hosts file is stored in the local environment, the definition works commonly regardless on LAN or mobile network. However, it is not easy to deal with in Android, and iPhone and iPad can't support it without jailbreak.

The main problem is that the domains to be resolved must be listed for each subdomain. As SecurrityTrails shows, the number of subdomains is too large for an individual to list them up on single Hosts file.

SecurityTrails
Subdomains of googlesyndication.com - SecurityTrails
Subdomains of outbrain.com - SecurityTrails
Subdomains of popin.cc - SecurityTrails
Subdomains of taboola.com - SecurityTrails
Subdomains of yimg.jp - SecurityTrails
Subdomains of yjtag.jp - SecurityTrails

Host files containing the blocked subdomains are distributed on various sites*2. These files can be reused, but they do not always contain the desired domain, and may contain subdomains that are not useful for blocking. They may also contain settings that direct you to inappropriate destinations.

A practical tactic would be to narrow down your subdomains. Looking at the ads that appear in "observation area", and try to register only the domains frequently appear in a hosts file. Following Pareto's law, blocking 20% of subdomains should block about 80% of ads in this area. Actually, in case of 2 domains below in SecurityTrails,

  • yimg.jp
  • yjtag.jp

Blocking only 3 subdomains below can block most ads from Yahoo! Japan.

0.0.0.0 s.yjtag.jp
0.0.0.0 im.c.yimg.jp yads.c.yimg.jp

Case - Yahoo! Fiance Japan
f:id:espio999:20210819001931p:plain
f:id:espio999:20210819001948p:plain

If wild card "*" is available for covering multiple subdomains, it makes operation and management of blocking domains simpler and easier. It is available in DNS.

DNS proxy

Applicable PC
Android
iPhone, iPad
Pros Commonly work for multiple devices
Easily register and manage domains
Cons Only for LAN, not for mobile network
dependency of DNS services

In DNS, wild card "*" is available as followings.

Case - Hosts file

0.0.0.0	vra.outbrain.com
0.0.0.0	vrp.outbrain.com
0.0.0.0	vrt.outbrain.com
0.0.0.0	widgets.outbrain.com
0.0.0.0	www.api.taboola.com
0.0.0.0	www.c2.taboola.com
0.0.0.0	www.cdn.taboola.com

Case - DNS

*.outbrain.com
*.taboola.com

A router would work as DNS proxy in LAN. Example, the home gateway lend by NTT has capability to forward DNS per specific domain.

f:id:espio999:20210819000055p:plain

DNS proxy can handle domains as

  • for specific domains, forward to ad block DNS service
  • for other domains, resolve as usual

AdGuard DNS*3 is one of ad block DNS services. Forwarding to such service, name resolution can be out sourced.
Example, forwarding 2 domains in "Case of DNS" above to AdGurard DNS, ads from Tabloola and Outbrain are disappeared as followings.

Case - IGN Japan
f:id:espio999:20210817235606p:plain
f:id:espio999:20210817235624p:plain

Case - togetter
f:id:espio999:20210817235712p:plain
f:id:espio999:20210817235735p:plain

However, all domains are not always blocked as expected. Example, forwarding "*.yimg.jp", and nslookup for 2 subdomains below returns global IP addresses, not 0.0.0.0.

  • im.c.yimg.jp
  • yads.c.yimg.jp

nslookup returns 0.0.0.0 for ”www.taboola.com”. It means AdGuard DNS recognizes taboola should be blocked, but not for yimg.

f:id:espio999:20210819000034p:plain

This is typical disadvantage from service dependency. Ad block DNS service doesn't always block specified domains strictly. Then, combination of technologies as required as

sieve domain level services as AdGuard
fine-meshed filter subdomain level Hosts

A service as AdGuard broadly blocks, and Hosts blocks subdomains that AdGuard couldn't catch. Combination of technologies as this can work maximum with minimum effort as followings.

Ad block by DNS proxy + Hosts
f:id:espio999:20210817235754p:plain
f:id:espio999:20210817235806p:plain
f:id:espio999:20210817235817p:plain
f:id:espio999:20210817235828p:plain

Sample - Domains forwarded to AdGurad

*.doubleclick.net
*.googlesyndication.com
*.googlevideo.com
*.i-mobile.co.jp
*.impact-ad.jp
*.logly.co.jp
*.microad.net
*.outbrain.com
*.popin.cc
*.taboola.com
*.taboolasyndication.com
*.yimg.jp
*.yjtag.jp


Local DNS

Applicable PC
Pros Can be stored in local environment
Easily register and manage domains
Cons Install and configuration per device

DNS proxy in a router is not portable. Blocking for most domains depends on services as AdGuard. If similar block capability is required for mobile network, followings would be degraded alternative.

  • block only with Hosts, without AdGuard
  • switch DNS to AdGuard in mobile network

Hosting DNS locally, it works not only as proxy, but also eliminate Hosts file. There are 2 options. The 1st option is the best solution, because it has no dependency on a service as AdGuard. Everything is under control of the users.

Option 1
Local DNS
return 0.0.0.0 for all specified domains
forward other domains to usual DNS
Options 2
Local DNS proxy
forward specified domains to AdGuard
forward other domains to usual DNS

In any case, Operation and management of domains are easier than Hosts file, because domains can be listed up with wild card. Even DNS is hosted locally, DNS communication within local environment should be established through Windows Defender Firewall with Advanced Security *4

Popular DNS for Windows 10 is as followings.

Popular DNS for Windows 10
www.isc.org
technitium.com
github.com