[minor] "precision" of $SECONDS

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[minor] "precision" of $SECONDS

Stephane Chazelas-8
$ time bash -c 'while ((SECONDS < 1)); do :; done'
bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 0.387 total

That can take in between 0 and 1 seconds. Or in other words,
$SECONDS becomes 1 in between 0 and 1 second after the shell was
started.

The reason seems to be because the shell records the value
returned by time() upon start-up and $SECONDS expands to
time()-that_saved_time. So, if bash is started at 10:00:00.999,
then $SECONDS will become 1 only a milisecond after startup
while if it's started at 10:00:01.000, $SECONDS will become 1 a
full second later.

IMO, it would be better if gettimeofday() or equivalent was used
instead of time() so that $SECONDS be incremented exactly one
second after start-up like ksh93 does.

mksh and zsh behave like bash (I'll raise the issue there as
well).

With zsh (like in ksh93), one can do "typeset -F SECONDS" to
make $SECONDS floating point, which can be used as a work around
of the "issue".

--
Stephane


Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

L A Walsh


Stephane Chazelas wrote:
> $ time bash -c 'while ((SECONDS < 1)); do :; done'
> bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 0.387 total
>
> That can take in between 0 and 1 seconds. Or in other words,
> $SECONDS becomes 1 in between 0 and 1 second after the shell was
> started.
The format you are using to display output of 'time' doesn't show
real time -- only CPU seconds.

Try:

TIMEFORMAT='%2Rsec %2Uusr %2Ssys (%P%% cpu)'

It shows the real, "clock" time, as well as
the standard % formula of cpu-secs/real-secs -- which can give
up to #Cores x 100"%" as value for %cpu  stretches the standard
def of "percent", but is at least more honest than grouping
all cpu's together as "1 cpu", and showing 100% usage of 1 core
on a 12-core machine as "8.3%".

>
> mksh and zsh behave like bash (I'll raise the issue there as
> well).
>
> With zsh (like in ksh93), one can do "typeset -F SECONDS" to
> make $SECONDS floating point, which can be used as a work around
> of the "issue".
>  
----
    With linux, one can read /proc/uptime to 100th's of a sec, or
use date to get more digits.  A middle of the road I used for
trace timing was something like:

function __age { declare ns=$(date +"%N"); declare -i
ms=${ns##+(0)}/1000000;
  printf "%4d.%03d\n" $SECONDS $ms
}
Then I add that in my PS4 prompt:
## for *relative* script exec speed, only!
##   (slows down normal speed significantly):
#export
PS4='>[$(__age)]${BASH_SOURCE:+${BASH_SOURCE[0]/$HOME/\~}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}>
'

On cygwin, calling 'date' was a deal breaker, so I just used /proc/uptime
to get the relative time down to centiseconds.

As you can see, I wanted the times
relative to the start of a given script, thus used SECONDS for that.

Given the overhead of calling 'date', the times printed are easily
20-50x slower
than w/o the time -- BUT, as the comment says above, it's good for
determining relative differences -- not only times for each command, but
also algorithmic differences, as the overhead is relatively constant.





Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

Stephane Chazelas-8
2016-02-25 03:03:41 -0800, Linda Walsh:

> Stephane Chazelas wrote:
> >$ time bash -c 'while ((SECONDS < 1)); do :; done'
> >bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 0.387 total
> >
> >That can take in between 0 and 1 seconds. Or in other words,
> >$SECONDS becomes 1 in between 0 and 1 second after the shell was
> >started.
> The format you are using to display output of 'time' doesn't show
> real time -- only CPU seconds.

It does. The last number (0.387 total) is the elapsed time in
the output of zsh (my interactive shell)'s "time" keyword. CPU
times are 0.39s user and 0.00s system totalling to 0.39s

Because it's a busy loop, CPU time is close to 100% (99%) so
elapsed and CPU time are roughly the same.

> Try:
>
> TIMEFORMAT='%2Rsec %2Uusr %2Ssys (%P%% cpu)'

That would be for bash. In anycase, bash does already include
the elapsed time in its default time output like zsh.

But the problem here is not about the time keyword, but about the
$SECONDS variable.

[...]
>    With linux, one can read /proc/uptime to 100th's of a sec, or
> use date to get more digits.  A middle of the road I used for
> trace timing was something like:
>
> function __age { declare ns=$(date +"%N"); declare -i
> ms=${ns##+(0)}/1000000;
>  printf "%4d.%03d\n" $SECONDS $ms
> }
[...]

I'm not sure how that gives you the time since startup.
Currently, if bash is started at

00:00:00.7

After 0.4 seconds (at 00:00:01.1), $SECONDS will be 1 (the "bug"
I'm raising here). "ms" will be 100, so you'll print 1.100
instead of 0.600. And with my suggested fix, you'd print 0.100.

[...]
> As you can see, I wanted the times
> relative to the start of a given script, thus used SECONDS for that.

Note that all of zsh, ksh93 and mksh have builtin support to get
elapsed time information with subsecond granularity.

zsh has:
  - $SECONDS: time since shell start. floating point after
    typeset -F SECONDS
  - $EPOCHSECONDS (unix time) (in zsh/datetime module)
  - $EPOCHREALTIME: same as floating point
  - zselect builtin to sleep with 1/100s granularity
    (in zsh/zselect module)
  - the "time" keyword, without a command prints CPU and real
    time for the shell and waited-for ancestors (and other
    getrusage statistics you can add with TIMEFMT)

ksh93 has:
  - $SECONDS: time since shell start. floating point after
    typeset -F SECONDS
  - EPOCHREALTIME=$(printf '%(%s.%N)T' now) for unix time as a
    float (note that you need a locale where the decimal
    separator is a period to be able to use that in ksh
    arithmetic expressions, or you need to replace that "." with
    a "," above).
  - builtin sleep with sub-second granularity

mksh has:
  - $EPOCHREALTIME: unix time as floating point (note however
  that mksh doesn't support floating point arithmetic).
  - builtin "sleep" command with sub-second granularity.

Similar features would be welcome in bash.

bash has "times" that gives you CPU time with sub-second
granularity. It's got a "printf %T" a la ksh93, but no %N, its
$SECOND is only integer (and currently has that issue discussed
here).

It does supports a $TMOUT with sub-second granularity though.
You can use that to sleep for sub-second durations if you find a
blocking file to read from. On Linux, that could be:

TMOUT=0.4 read < /dev/fd/1 | :

but that still means forking processes.

--
Stephane

Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

Stephane Chazelas-8
2016-02-25 13:18:17 +0000, Stephane Chazelas:
[...]

> > function __age { declare ns=$(date +"%N"); declare -i
> > ms=${ns##+(0)}/1000000;
> >  printf "%4d.%03d\n" $SECONDS $ms
> > }
> [...]
>
> I'm not sure how that gives you the time since startup.
> Currently, if bash is started at
>
> 00:00:00.7
>
> After 0.4 seconds (at 00:00:01.1), $SECONDS will be 1 (the "bug"
> I'm raising here). "ms" will be 100, so you'll print 1.100
> instead of 0.600. And with my suggested fix, you'd print 0.100.
             0.400

Sorry, meant 0.400 above.

--
Stephane


Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

Chet Ramey
In reply to this post by Stephane Chazelas-8
On 2/25/16 8:18 AM, Stephane Chazelas wrote:

> Similar features would be welcome in bash.
>
> bash has "times" that gives you CPU time with sub-second
> granularity. It's got a "printf %T" a la ksh93, but no %N, its
> $SECOND is only integer (and currently has that issue discussed
> here).

Because bash doesn't have floating point arithmetic.  There's no
real reason to have $SECONDS in a format you can't use to perform
calculations.

Bash's %T implementation doesn't have %N because it uses the libc
strftime(3), and as far as I know, no strftime provides it.  I assume
that ksh93 implements it internally as part of libast.

Bash doesn't have a `sleep' builtin at all, but there is a loadable
sleep builtin that offers sub-second granularity.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    [hidden email]    http://cnswww.cns.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

Stephane Chazelas-8
2016-02-25 10:48:51 -0500, Chet Ramey:
[...]
> Because bash doesn't have floating point arithmetic.

Yes, makes sense. mksh having $EPOCHREALTIME floating point even
though it doesn't have floating point arithmetic does sound
weird.

Any plan of adding floating point arithmetic support to bash by
the way?

> There's no
> real reason to have $SECONDS in a format you can't use to perform
> calculations.

That could be done with an extra $NANOSECONDS variable, but then
that wouldn't be reliable as in now=$SECONDS.$NANOSECONDS,
$SECONDS and $NANOSECONDS could be expanded at different seconds
(if run for instance at 00:00:00.9999999).

A printf '%(sec=%s nsec=%N)T' -1 wouldn't have the problem though.

> Bash's %T implementation doesn't have %N because it uses the libc
> strftime(3), and as far as I know, no strftime provides it.  I assume
> that ksh93 implements it internally as part of libast.
[...]

Probably. Note that GNU date also has a %N and doesn't use
strftime either. strftime taking a struct tm can't have
subseconds anyway.

--
Stephane

Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

L A Walsh
In reply to this post by Stephane Chazelas-8


Stephane Chazelas wrote:
> 2016-02-25 03:03:41 -0800, Linda Walsh:
>
>  
>> Stephane Chazelas wrote:
>>    
>>> $ time bash -c 'while ((SECONDS < 1)); do :; done'
>>> bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 0.387 total
>>>      
----
    Sorry I took "cpu xxx total" to be the total cpu time.  Silly me.
(I do believe you, just the display format could be more clear).

>
>
>  
>> Try:
>>
>> TIMEFORMAT='%2Rsec %2Uusr %2Ssys (%P%% cpu)'
>>    
>
> That would be for bash. In anycase, bash does already include
> the elapsed time in its default time output like zsh.
>  
---
but not as clearly, IMO... ;-)

> But the problem here is not about the time keyword, but about the
> $SECONDS variable.
>  
---
    I realize that.

> [...]
>  
>>    With linux, one can read /proc/uptime to 100th's of a sec, or
>> use date to get more digits.  A middle of the road I used for
>> trace timing was something like:
>>
>> function __age { declare ns=$(date +"%N"); declare -i
>> ms=${ns##+(0)}/1000000;
>>  printf "%4d.%03d\n" $SECONDS $ms
>> }
>>    
> [...]
>
> I'm not sure how that gives you the time since startup.
>  
---
    The time since the bash script startup.
I was guessing that SECONDS recorded the integer value of 'seconds'
at the start of the script.  Thus it shows later times as the
later recorded time in seconds - the original time in seconds -- or
at least that would match current behavior -- initial "seconds"
param from gettime, (ignoring the nano or centi secs, depending on call).

going from an integer value of the time
at start
> Currently, if bash is started at
>
> 00:00:00.7
>  
----
    Well, ok, but time calls usually give back
seconds from 1970, with some giving back another param for
centi, or more modern calls giving back 2nd parm for nano's.

Theoretically, bash could never start when seconds=0, unless
it was started in 1970...  But I'm guessing you are using clock
time, whereas I was using the time from the start of the script.

I.e. @ start of script, SECONDS gets the # secs since 1970,
and (if done at the same time, the date call for nanosecs) would
be the #nanos above that number of secs.


> After 0.4 seconds (at 00:00:01.1), $SECONDS will be 1 (the "bug"
> I'm raising here). "ms" will be 100, so you'll print 1.100
> instead of 0.600. And with my suggested fix, you'd print 0.100.
>  
---
At bash startup, I'll see 0 seconds, and 7000,000 nanosecs.
after .4 secs, Ill see 1 sec & 1000,000 nanosecs.

So diff would be 1.1=0.7 = .4secs = correct answer, no?


>
> Note that all of zsh, ksh93 and mksh have builtin support to get
> elapsed time information with subsecond granularity.
>  
That's not a POSIX requirement, and bash is hardly an ideal tool
to need or rely on sub-second granularity, especially since
it doesn't process signals in real-time, but only upon
a keypress in readline.



Reply | Threaded
Open this post in threaded view
|

Re: [minor] "precision" of $SECONDS

Chet Ramey
In reply to this post by Stephane Chazelas-8
On 2/25/16 12:20 PM, Stephane Chazelas wrote:

> 2016-02-25 10:48:51 -0500, Chet Ramey:
> [...]
>> Because bash doesn't have floating point arithmetic.
>
> Yes, makes sense. mksh having $EPOCHREALTIME floating point even
> though it doesn't have floating point arithmetic does sound
> weird.
>
> Any plan of adding floating point arithmetic support to bash by
> the way?

It's not on the short-term feature list.


>> Bash's %T implementation doesn't have %N because it uses the libc
>> strftime(3), and as far as I know, no strftime provides it.  I assume
>> that ksh93 implements it internally as part of libast.
> [...]
>
> Probably. Note that GNU date also has a %N and doesn't use
> strftime either. strftime taking a struct tm can't have
> subseconds anyway.

GNU date uses its own time-string-formatting function, derived from
strftime and extended to include things like %N.

Bash includes Aharon Robbins's open-source strftime for those systems
that don't have it; I suppose I could extend that and use it
unconditionally.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    [hidden email]    http://cnswww.cns.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

[repost] "precision" of $SECONDS

Stephane Chazelas-8
In reply to this post by Stephane Chazelas-8
That's a a re-post of a bug report I raised a few years ago and
comes back now and then in various Q&A sites I participate in.

The discussions kind of trailed off last time.
https://www.mail-archive.com/bug-bash@.../msg17783.html

2016-02-24 15:16:41 +0000, Stephane Chazelas:

> $ time bash -c 'while ((SECONDS < 1)); do :; done'
> bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 0.387 total
>
> That can take in between 0 and 1 seconds. Or in other words,
> $SECONDS becomes 1 in between 0 and 1 second after the shell was
> started.
>
> The reason seems to be because the shell records the value
> returned by time() upon start-up and $SECONDS expands to
> time()-that_saved_time. So, if bash is started at 10:00:00.999,
> then $SECONDS will become 1 only a milisecond after startup
> while if it's started at 10:00:01.000, $SECONDS will become 1 a
> full second later.
>
> IMO, it would be better if gettimeofday() or equivalent was used
> instead of time() so that $SECONDS be incremented exactly one
> second after start-up like ksh93 does.
>
> mksh and zsh behave like bash (I'll raise the issue there as
> well).
>
> With zsh (like in ksh93), one can do "typeset -F SECONDS" to
> make $SECONDS floating point, which can be used as a work around
> of the "issue".
[...]

Note that since then the corresponding bugs in mksh and zsh have
been fixed.

--
Stephane