That does seem intuitve but it's not necessarily true. It depends on the profile of the bullet velocity vs distance over the trajectory to the target. Chapter 7, "The effect of wind on flat fire trajectories" in Robert McCoy's book "Modern Exterior Ballistics" has a discussion of the sensitivity to wind versus downrange distance. It's a good read. This graph gives an example of a case where the location of peak wind sensitivity is not at the shooter's location.
Good graph.
The 1000-yard plot is especially interesting. 30 M2 ball throws a bullet of about 150 grains at about 2800 fps. (I use "about" because different references say different things and I don't want to get bogged down in minor irrelevancies.) The chart shows that said bullet is most susceptible to wind deflection at about 400-500 yards. At that range, the bullet speed has dropped to about 1800 fps or so. The exact figure isn't important; what's important to understanding wind drift is the neighborhood we're now in.
Wind drift isn't a function of speed or time of flight. It's a function of the rate of change of bullet velocity. As a bullet enters the transsonic range, it sheds speed more quickly and drifts more. The phenomenon starts around 2300 fps. As a bullet slows down below that speed, the amount of wind drift increases more than time of flight would seem to suggest. The effect is worst at about 1700 fps. Then the process reverses itself and the bullet sheds velocity less quickly as speed drops further. Thus, wind drift then becomes
less as the bullet slows from 1700 to about 900 fps.
Wind drift is about the same at 900 fps as it is at 2300 fps. In between is worse than the extremes.
So your chart shows exactly that. The bullet impact is more influenced by the wind at the 400-500 yard distance than either before or after. If the target is 600 yards away, this may not make much difference. But if the target is 1000 yards away, the deflection that occurred halfway to the target has much more time to push the bullet away from center.
If long range shooters wanted to minimize wind drift and didn't care about the ridiculous elevation problems that would crop up, they'd be better off launching 30 caliber 300-grain VLDs at 950 feet per second.
Obviously, that's not going to happen. But it does go to prove something I constantly harp about: Studying ballistics is taking a walk in the land of the weird.