| Re: Asynchronous FTP Upload [message #172128 is a reply to message #172113] |
Mon, 31 January 2011 12:29   |
The Natural Philosoph
Messages: 993
Registered: September 2010
Karma:
|
Senior Member |
|
|
Peter H. Coffin wrote:
> On Sun, 30 Jan 2011 17:06:40 +0000, The Natural Philosopher wrote:
>> Peter H. Coffin wrote:
>>> On Sun, 30 Jan 2011 15:48:29 +0100, Luuk wrote:
>>>> On 30-01-11 15:20, Jerry Stuckle wrote:
>>>> > Not really. You're not going to get 1gb/sec. or even 200mb/sec. from
>>>> > the disk drive, especially not continuously. So even if the download
>>>> > speed on the other end is 200mb/sec, that's still not going to be a
>>>> > limiting factor.
>>>> >
>>>> > And forcing the disk to pull data from several different files on the
>>>> > disk will slow disk overall disk access even more, especially if the
>>>> > files are contiguous.
>>>> But if the files are send to 500 hosts, the file might be in cache, if
>>>> enough memory is available, which should speed up disk-access again.. ;)
>>> That'd be likely true for for a Very Large system, but I'd not bet on it
>>> for the "hundreds of megabytes" original situation unles it's a
>>> completely dedicated system/cache. There's still other processes going
>>> on that are going to end up with their own stuff.
>>>
>> of course it will be cached if there is adequate memory to do it. If
>> there isn't, add more.
>
> Sometimes that's a solution.
>
>> At est on Linux EVERYTHING is cached to the memory limit: Only if a
>> process needs more real physical ram than is available, will the
>> existing disk cache buffers be flushed.
>
> That's kinda the point. EVERYTHING gets cached, without much attention
> being paid to what's going to be needed again and when. (At least, that
> was the situation when I last looked at it; there wasn't any way to tell
> the OS "This file is more important to keep in cache than that one.")
The algorithms are fairly smart though. LRU and all that. If you cant
cache the file, at least cache the directory.
But n this case if the SAME file is being pushed multiple times it is
almost guaranteed to be in cache.
> Which on a busy system means you have a very FULL cache, but not
> necessarily that the right parts of a Large File are going to be there.
> The next bit needed to send to a download may well have already been
> purged because some log file wanted its space in the cache.
Not if there is more than a minimum amount of memory.
the dirty buffers will tend to be cleaned by sorting them for most
efficient disk access, and clean buffers can be simply freed up with no
disk access at all.
Whatever is then re-cached will tend to be what was last used, and what
is hardest to re-load in terms of disk access time.
If the system is actually being used by live users on most or all
processes, that's very close to optimal: If you are short of physical
RAM then someone is going to suffer: Your argument is then about making
sure its not YOUR application, rather than someone elses.
>
> writing a dedicated downloader app that can allocate that much memory
> for the whole file,
Does not ensure its actually in RAM either.
The OS is quite capable of paging out such RAM if another app needs it.
I think you are somehow not quite understanding how a modern OS uses
memory. In general you cannot hard allocate RAM at all. Only kernel
processes have that privilege, IIRC.
This is one of the reasons why RTOS designs exist: If you need to
guarantee response times you have to return to a more simplified OS
design, running very specific code.
> then share that memory explicitly with many little
> clones of itself will ensure that purging needed bits doesn't happen.
It wont..
> And a tool like that can probably come very, very close to saturating
> the outbound link, especially if there's something that's being semi
> intelligent about managing the spawed processes. For example, it could
> keep track of the average speeds to various sites and when it sees that
> it has X kbps left between expected link speed and what's currently
> being sent, and it can then select another site to send to based on it
> being the highest historical rate that fits into the remaining bandwidth
> window.
>
> But this is, I think getting a little too far into the design aspect,
> though I think it's probably still within PHP's capacity.
>
No, its completely outside of it actually.
Its well into device driver or privileged daemon territory.
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
]