From roberto.foglietta at gmail.com Sun Sep 3 15:57:27 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 3 Sep 2023 17:57:27 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm Message-ID: Hi, just for your information: pidof $string pgrep $string are sensitively slower than this one fgrep "$string" /proc/[1-9]*/comm | cut -d/ -f3 in particular when taskset -pc 0 $$ is used, but also in general. I can accept that my system is the root cause of this issue. Moreover, it is not the only anomaly that I have noticed. f() { local pippo=$(command1) ; echo $pippo; } is sensitively slower than f() { local pippo; pippo=$(command1) ; echo $pippo; } Best regards, R- From bb at npl.de Mon Sep 4 16:53:12 2023 From: bb at npl.de (Bastian Bittorf) Date: Mon, 4 Sep 2023 16:53:12 +0000 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: Message-ID: <20230904165312.tct4vx4h67qasttz@email> can you please give us a hint, what makes your system special? maybe several 1000 processes? can you please show, how you measure the time differences on your 2nd problem? thanks & bye, Bastian Bittorf From roberto.foglietta at gmail.com Mon Sep 4 19:52:11 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 4 Sep 2023 21:52:11 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: <20230904165312.tct4vx4h67qasttz@email> References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Mon, 4 Sept 2023 at 18:53, Bastian Bittorf wrote: > > can you please give us a hint, > what makes your system special? > maybe several 1000 processes? Hi Bastian, on that system there are 8 Arm 64bit cores 4x1.8GHz + 4x2.0Ghz put under conservative frequency governors usually sleeping but sometimes running at their lowest frequency 300MHz and I have no clue what's going to make my system "special". Someone says that it is the man behind the screen but they are rare while almost others ask for statistics. Do you wish to receive statistics? ;-) Just few facts: - moving from local var=$(command) to local var; car=$(commad) - avoid to use pidof and pgrep but grep over /proc - source instead of calling a script The script time to be ready for the user interaction drops from 1.7s to 360ms. Usually, I consider a anything above 250ms a latency to fix for a smooth UI/UX but considering it is a telnet app, I am keen to accept 360ms and I will take care of this aspect when I will transform the script which delivers a menu (text interface by telnet) in a server that exchange data by a socket / fifo and source others scripts when an action is requested by user on demand. Possibly, I will straightforwardly go for a CGI-BIN httpd in order to use a common web interface. However, I love the idea of keeping the telnet menu, it makes me feel so nostalgic and old.. ROTFL Best regards, R- From roberto.foglietta at gmail.com Mon Sep 4 20:54:47 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 4 Sep 2023 22:54:47 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Mon, 4 Sept 2023 at 21:52, Roberto A. Foglietta wrote: > > On Mon, 4 Sept 2023 at 18:53, Bastian Bittorf wrote: > > > > can you please give us a hint, > > what makes your system special? > > maybe several 1000 processes? > > Hi Bastian, > > on that system there are 8 Arm 64bit cores 4x1.8GHz + 4x2.0Ghz put > under conservative frequency governors usually sleeping but sometimes > running at their lowest frequency 300MHz and I have no clue what's > going to make my system "special". Someone says that it is the man > behind the screen but they are rare while almost others ask for > statistics. Do you wish to receive statistics? ;-) s/behind/in front of/ but usually behind because I use the network rather than a physical keyboard to be connected to the device. Moreover, the sleep 1 sometimes sleeps for 2 seconds but can also sleep 7 seconds (I saw a 16s but I do not believe it). While sleep 60 is 1:1000 precise and pause() { sleep 500000*${1:-1}; } always gives reasonable precise delays. Is the ratio 1:7 because 2GHz / 300Mhz? I did not investigate this problem yet but just let the system collect statistics about it. Therefore, sleep 1 sleeps for 2 seconds, is a common fact not a corner case. The sleep and usleep are both applets of busybox, the last stable branch compiled static against glibc. Best regards, R- From roberto.foglietta at gmail.com Tue Sep 5 03:19:14 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 5 Sep 2023 05:19:14 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Mon, 4 Sept 2023 at 22:54, Roberto A. Foglietta wrote: > > On Mon, 4 Sept 2023 at 21:52, Roberto A. Foglietta > wrote: > > > > On Mon, 4 Sept 2023 at 18:53, Bastian Bittorf wrote: > > > > > > can you please give us a hint, > > > what makes your system special? > > > maybe several 1000 processes? > > > > Hi Bastian, > > > > on that system there are 8 Arm 64bit cores 4x1.8GHz + 4x2.0Ghz put > > under conservative frequency governors usually sleeping but sometimes > > running at their lowest frequency 300MHz and I have no clue what's > > going to make my system "special". Someone says that it is the man > > behind the screen but they are rare while almost others ask for > > statistics. Do you wish to receive statistics? ;-) > > s/behind/in front of/ but usually behind because I use the network > rather than a physical keyboard to be connected to the device. > It happens only when a script is executed by telnetd (a busybox app). However, an ash login shell spawned by that script parented by telnetd does not show the issue. The busybox is configured to not fork apps, compiled with -02 and ash for speed. The telnetd is set to run on CPU0 only by taskset. The busybox configuration is here: https://github.com/robang74/sailfish-os-busybox/blob/e03f839f9ce267279e18c6b7b9bc313e74e84106/rpm/busybox-static.config The issue about sleep 1 seems fixed by now. Some default values that I found in the kernel scheduler configuration were completely out of range, IMHO. Now, the maximum drift on sleep 1 that I have seen in stats is below 2%. In others nights stats less than 5%. I do not trust this stats very much because I have not stressed-out the system but the original system was not even able to set CPUs to sleep without the need to awake them immediately after and so on, so on, so on... Best regards, R- From roberto.foglietta at gmail.com Wed Sep 6 00:38:58 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 6 Sep 2023 02:38:58 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Mon, 4 Sept 2023 at 21:52, Roberto A. Foglietta wrote: > > Just few facts: > > - moving from local var=$(command) to local var; car=$(commad) > - avoid to use pidof and pgrep but grep over /proc > - source instead of calling a script > > The script time to be ready for the user interaction drops from 1.7s > to 360ms. I have managed to replicate the two systems behaviours: SLOW 1.7s - local var=$(command) - pidof command1 command2 command3 - telnetd running on CPU0 only script as login handler - busybox static with NOFORK option enabled FASTEST 360ms - local var; var=$(command) - pgrep -E "command1|command2|command3" /proc/[1-9]*/comm - telnetd running on CPU0 only sourcing a script as login handler - taskset -pc 0-7 at the beginning of the script run by telnetd - busybox static with NOFORK option enabled OTHERS (from 450ms to 600ms) - local var=$(command) - pidof command1 command2 command3 - telnetd running on CPU0 only script as login handler - busybox static with NOFORK option disabled - local var=$(command) - pidof command1 command2 command3 - telnetd running on CPU0 only script as login handler - taskset -pc 0-7 at the beginning of the script run by telnetd - busybox static with NOFORK option enabled CONCLUSION - busybox ash (standalone static) NOFORK running on a single CPU is a bottleneck not a script interpreter. - busybox ash (standalone static) NOFORK running on a multiple CPU is faster than the forking version as expected. I did not verify the code yet. However, I think that it is necessary to introduce a test in the NOFORK code for which as long as busybox is running on a single CPU, it forks otherwise follow the default policy chosen at the building time. I hope this helps, R- From lengtt258 at gmail.com Wed Sep 6 04:27:27 2023 From: lengtt258 at gmail.com (Leng TouTou) Date: Wed, 6 Sep 2023 12:27:27 +0800 Subject: memory increase caused by mounting failure In-Reply-To: References: Message-ID: Hi? I used the following command to mount repeatedly about 10,000 times $ mount -t squashfs -o loop,thread=0 image /mnt/tmp This command will not mount successfully, then I found that the memory used by slab increased by about 10000k. So is this normal? In the Busybox source code, I saw that when the mount fails, del_loop uses LOOP_CLR_FD for cleanup. Does LOOP_CLR_FD simply disconnect the loop device from the image, leaving the loop device still available for reuse? Or if I want to completely release a loop device and its occupied memory, do I need to use LOOP_CTL_REMOVE? Best regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.foglietta at gmail.com Wed Sep 6 09:38:38 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 6 Sep 2023 11:38:38 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Wed, 6 Sept 2023 at 02:38, Roberto A. Foglietta wrote: > > I did not verify the code yet. However, I think that it is necessary > to introduce a test in the NOFORK code for which as long as busybox is > running on a single CPU, it forks otherwise follow the default policy > chosen at the building time. A single test is NOT a statistic nor a benchmark, just an example of what I am seeing: redfishos:~ # taskset -pc $$ pid 25454's current affinity list: 0-3 redfishos:~ # a=$(date +%s%N); pidof yamui; b=$(date +%s%N); echo $((b-a)) 766 102291042 We can set this number as reference 100% to compare the others executions time in a relative way. redfishos:~ # a=$(date +%s%N); grep -E "yamui$" /proc/[1-9]*/comm | cut -d/ -f3; b=$(date +%s%N); echo $((b-a)) 766 77236302 redfishos:~ # echo $((77236302 / 1022910))% 75% redfishos:~ # a=$(date +%s%N); pgrep -x yamui; b=$(date +%s%N); echo $((b-a)) 766 89810105 redfishos:~ # echo $((89810105 / 1022910))% 87% Moreover the execution of this function (preloaded into the environment) gives the same time consumption of the grep -E command without any appreciable delay. comm_pid() { local i str test -n "${1:-}" || return 1 for i in "$@"; do str="$i${str:+|$str}"; done grep -E "$str" /proc/[1-9]*/comm | cut -d/ -f3 } redfishos:~ # a=$(date +%s%N); comm_pid yamui$; b=$(date +%s%N); echo $((b-a)) 766 76270209 redfishos:~ # echo $(( 76270209 / 1022910 ))% 74% This means that not only two commands in the pipe are faster than an applet but a simple function is faster than an applet. You can imagine my surprise. I hope this helps, R- From roberto.foglietta at gmail.com Wed Sep 6 10:09:30 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 6 Sep 2023 12:09:30 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Wed, 6 Sept 2023 at 11:38, Roberto A. Foglietta wrote: > Moreover the execution of this function (preloaded into the > environment) gives the same time consumption of the grep -E command > without any appreciable delay. To be fair the function should return the status like pidof and pgrep, therefore a grep . should be added in pipe to the last command comm_pid() { local i str test -n "${1:-}" || return 1 for i in "$@"; do str="$i${str:+|$str}"; done grep -E "$str" /proc/[1-9]*/comm | cut -d/ -f3 | grep . } > > redfishos:~ # a=$(date +%s%N); comm_pid yamui$; b=$(date +%s%N); echo $((b-a)) > 766 > 76270209 > redfishos:~ # echo $(( 76270209 / 1022910 ))% > 74% > However, execution time does not change so much redfishos:~ # a=$(date +%s%N); comm_pid yamui$; b=$(date +%s%N); echo $((b-a)); 766 79261041 redfishos:~ # echo $(( 79261041 / 1022910 ))% 77% Best regards, R- From roberto.foglietta at gmail.com Wed Sep 6 12:16:45 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 6 Sep 2023 14:16:45 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Wed, 6 Sept 2023 at 12:09, Roberto A. Foglietta wrote: > However, execution time does not change so much Instead when multiple processes names are passed as arguments, the execution time gap increases even more because pidof execution time depends on the number of arguments while the function listed here below does not. Therefore, I think that there is an algorithm change to plan for the pidof applet that will move it from O(N) to O(1). comm_pid() { grep -nE "$(echo $* | tr ' ' '|')" /proc/[1-9]*/comm | cut -d/ -f3 | grep . } redfishos:~ # a=$(date +%s%N); pidof sshd dropbear telnetd httpd yamui; b=$(date +%s%N); echo $((b-a)) 678 673 675 725 456542916 redfishos:~ # a=$(date +%s%N); comm_pid sshd dropbear telnetd httpd yamui; b=$(date +%s%N); echo $((b-a)) 673 675 678 680 725 95683125 redfishos:~ # echo $(( 456542916 / 956831 )) 477 redfishos:~ # a=$(date +%s%N); comm_pid sshd dropbear telnetd httpd yamui ash; b=$(date +%s%N); echo $((b-a)) 1866 438 673 675 678 680 725 95632552 redfishos:~ # a=$(date +%s%N); pidof sshd dropbear telnetd httpd yamui ash; b=$(date +%s%N); echo $((b-a)) 678 673 675 725 1866 548829427 redfishos:~ # echo $(( 548829427 / 956326 )) 573 I hope this helps, R- From roberto.foglietta at gmail.com Wed Sep 6 16:59:02 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 6 Sep 2023 18:59:02 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Wed, 6 Sept 2023 at 14:16, Roberto A. Foglietta wrote: > > On Wed, 6 Sept 2023 at 12:09, Roberto A. Foglietta > wrote: > > > However, execution time does not change so much > > Instead when multiple processes names are passed as arguments, the > execution time gap increases even more because pidof execution time > depends on the number of arguments while the function listed here > below does not. Therefore, I think that there is an algorithm change > to plan for the pidof applet that will move it from O(N) to O(1). I push further my investigation to check some details that might have influenced it: - replicate the basic pidof applet functionality for a fair comparison - account the time date applet takes to give the output Here we are: == SCRIPT == comw_pid() { local str # RAF: the /proc/$pid/comm contains the first 15 chars + ending \n str=$(echo -n $*| tr ' ' '\n'| sed -e "s,\(.\{15\}\).*,^\\1$,"| tr '\n' '|') str=$(grep -nE "$str" /proc/[1-9]*/comm 2>/dev/null | cut -d/ -f3) echo $str | grep --color=never . } # The str and datecmd values can change among systems dtc=$(for i in $(seq 1 100); do a=$($datecmd +%s%N); b=$($datecmd +%s%N); echo $((b-a)); done); str=$(echo "$dtc" | wc -l; echo "$dtc" | sort -n | head -n1; echo "$dtc" | sort -nr | head -n1) printf "tries: %d, min: %d, max: %d\n" $str; min=$(echo "$dtc" | sort -n | head -n1) a=$($datecmd +%s%N); comw_pid $procstr; b=$($datecmd +%s%N); dta=$((b-a-min)); c=$($datecmd +%s%N); pidof $procstr; d=$($datecmd +%s%N); dtb=$((d-c-min)); echo exec.times: $dta ns, $dtb ns, ratio: $(( (dtb*100) / dta ))% echo "pidof is $(readlink $(which pidof))" == RESULTS == Running on RedFish OS on busybox ash procstr="yamui yamui-powerkey yamui-screensaverd dropbear telnetd sshd sshd.bin httpd"; datecmd=date tries: 100, min: 4136563, max: 5006771 19681 673 675 680 725 725 680 673 19681 675 exec.times: 117145936 ns, 714783020 ns, ratio: 610% pidof is /sbin/busybox-static Running on Ubuntu 22.04 LTS busybox ash procstr="bash nautilus gedit dnsmasq thermald snapd hello world" datecmd=/bin/date tries: 100, min: 1610318, max: 3667151 1148885 1243481 1262 1271 14404 14447 14468 14489 1505 1508232 15566 255975 256074 256157 27375 3891 3892 432851 472771 51819 51823 679241 679272 95110 1508232 1243481 679272 679241 472771 432851 256157 256074 255975 95110 51819 27375 15566 14489 14468 14447 14404 1148885 51823 3892 3891 1505 1271 1262 exec.times: 33390016 ns, 49760852 ns, ratio: 149% pidof is /sbin/killall5 The two strings are equivalent: 6 existing procs + 2 words == == == == As far as I can understand, these data do not contradict my previous conclusion but sustain it even further. Moreover, also sysvinit-utils: /sbin/killall5 seems less performant than a script with a O(1) approach. Improving the pidof in busybox of 6x times (at least) means being 4x faster instead of being 2x slower than sysvinit-utils applet, by a rule of thumb. Best regards, R- From bb at npl.de Thu Sep 7 07:44:27 2023 From: bb at npl.de (Bastian Bittorf) Date: Thu, 7 Sep 2023 07:44:27 +0000 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: <20230907074427.j23zmyp6rpzlitxv@email> On Wed, Sep 06, 2023 at 02:38:58AM +0200, Roberto A. Foglietta wrote: > On Mon, 4 Sept 2023 at 21:52, Roberto A. Foglietta > wrote: > > > > Just few facts: > > > > - moving from local var=$(command) to local var; car=$(commad) > > - avoid to use pidof and pgrep but grep over /proc > > - source instead of calling a script > > > > The script time to be ready for the user interaction drops from 1.7s > > to 360ms. > > I have managed to replicate the two systems behaviours: I understand your commands, but i do not understand how you are measuring the time, see this: root at slow-embedded:~ 8( time grep -E "command1|command2|command3" /proc/[1-9]*/comm Command exited with non-zero status 1 real 0m 0.00s user 0m 0.00s sys 0m 0.00s It is so fast, how do you measure? bye, Bastian From roberto.foglietta at gmail.com Fri Sep 8 01:46:09 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Fri, 8 Sep 2023 03:46:09 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: <20230907074427.j23zmyp6rpzlitxv@email> References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> Message-ID: On Thu, 7 Sept 2023 at 09:44, Bastian Bittorf wrote: > I understand your commands, but i do not understand > how you are measuring the time, The busybox time does not work for functions. A function is needed to replicate the pidof basic behaviour otherwise the comparison can be considered unfair. redfishos:/rootfs # a=$(date +%s%N); time grep -E "command1|command2|command3" /proc/[1-9]*/comm; b=$(date +%s%N); c=$(date +%s%N); echo $(( (b-a) - (c-b) )) Command exited with non-zero status 1 real 0m 0.05s user 0m 0.01s sys 0m 0.03s 80300209 redfishos:/rootfs # a=$(date +%s%N); time pidof command1 command2 command3; b=$(date +%s%N); c=$(date +%s%N); echo $(( (b-a) - (c-b) )) Command exited with non-zero status 1 real 0m 0.28s user 0m 0.02s sys 0m 0.23s 294661666 redfishos:/rootfs # min=$(for i in $(seq 1 100); do b=$(date +%s%N); c=$(date +%s%N); echo $(( c-b )); done | sort -g | head -n1); echo $min 4220937 redfishos:/rootfs # a=$(date +%s%N); time grep -E "command1|command2|command3" /proc/[1-9]*/comm; b=$(date +%s%N); echo $(( (b-a) - min )) Command exited with non-zero status 1 real 0m 0.05s user 0m 0.00s sys 0m 0.04s 81258907 redfishos:/rootfs # a=$(date +%s%N); time pidof command1 command2 command3; b=$(date +%s%N); echo $(( (b-a) - min )) Command exited with non-zero status 1 real 0m 0.28s user 0m 0.02s sys 0m 0.23s 290395104 However, using time with grep and pidof for a confrontation still brings the same result. The grep is O(1) while the pidof is O(N). cmd=""; for i in $(seq 0 9); do cmd="${cmd:-} command$i"; a=$(date +%s%N); time pidof $cmd 2>&1 | grep real; b=$(date +%s%N); echo $(( (b-a) - min )); done real 0m 0.09s 106920365 real 0m 0.18s 198038438 real 0m 0.28s 291350782 real 0m 0.37s 388994896 real 0m 0.46s 479455261 real 0m 0.55s 564525573 real 0m 0.64s 658862240 real 0m 0.74s 750989739 real 0m 0.83s 843974480 real 0m 0.92s 935911458 redfishos:/rootfs # echo $cmd | tr ' ' '|' command0|command1|command2|command3|command4|command5|command6|command7|command8|command9 redfishos:/rootfs # time grep -E "command1" /proc/[1-9]*/comm 2>&1 | grep real real 0m 0.05s redfishos:/rootfs # time grep -E $(echo $cmd | tr ' ' '|') /proc/[1-9]*/comm 2>&1 | grep real real 0m 0.05s This is a matter of fact whether grep vs pidof for a single argument - the most probable case - would be faster or not. Moreover, grep shows to be nearly 2x faster also with a single argument. While with 4 arguments is 7x (mine estimation was nearly 5x) faster and with 5 arguments is 9x faster (mine estimation was nearly 6x). This because time showed us that grep is 1.6x faster than pidof when a single argument is passed while my way - a conservative way of taking times - was starting when n=1 with a 1:1 ratio. The two ways to take measures showed a difference of 7 - 18% when n(args) = 1 and 0.7 - 1.7% when n(args) = 10. This clearly a linear regression which tells us that the main source of error stays in the accuracy of the $min value which is an estimation. Therefore err(n) = err(min) / n. Fortunately, this is a conservative way of taking measures because I used the min value of 100. Conservative, it means that I could have not appreciated a small difference in performances between pidof and grep or a small difference in O(same). Such a small difference in performance would not have attracted my attention, in the first place and I would not have started to investigate the issue using statistics. Hence, the HP of seeing a wide difference in performance or in O(different) was acceptable from the beginning. More in general, how to take measures and dealing with errors and statistics are not common skills among IT guys unless they have a degree/bachelor in applied physics (or a theoretical physic with laboratories). Best regards, R- From roberto.foglietta at gmail.com Fri Sep 8 02:32:38 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Fri, 8 Sep 2023 04:32:38 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> Message-ID: On Fri, 8 Sept 2023 at 03:46, Roberto A. Foglietta wrote: > The two ways to take measures showed a difference of 7 - 18% when > n(args) = 1 and 0.7 - 1.7% when n(args) = 10. This clearly a linear > regression which tells us that the main source of error stays in the > accuracy of the $min value which is an estimation. Therefore err(n) = > err(min) / n. Fortunately, this is a conservative way of taking > measures because I used the min value of 100. Conservative, it means > that I could have not appreciated a small difference in performances > between pidof and grep or a small difference in O(same). Such a small > difference in performance would not have attracted my attention, in > the first place and I would not have started to investigate the issue > using statistics. Hence, the HP of seeing a wide difference in > performance or in O(different) was acceptable from the beginning. After all, time is not always precise enough to determine a measure: redfishos:/rootfs # time date +%s%N 1689574011301750356 real 0m 0.00s for i in $(seq 1 100); do date +%s%N; done | time cat >/dev/null real 0m 0.45s data=$(for i in $(seq 1 100); do a=$(date +%s%N); b=$(date +%s%N); echo $((b-a)); done | sort -n); echo "$data" | head -n1; echo "$data" | tail -n1; let avg=$(echo "$data" | tr '\n' '+')0; echo $(( (avg+50) / 100 )) 4233021 5255000 4475716 Average is 4475716 ns = 4475 us while 0.45s = 45 ms = 450000 us therefore 45000 us / 100 = 4500 us. These two ways of determining the $min values give compatible results. However, the value that reduces the gap between the time and date difference measures is the max not the average. max=5255000; cmd=""; for i in $(seq 0 9); do cmd="${cmd:-} command$i"; a=$(date +%s%N); time pidof $cmd 2>&1 | grep real; b=$(date +%s%N); echo $(( (b-a) - max )); done real 0m 0.09s 107959792 real 0m 0.18s 197792917 real 0m 0.28s 289114635 real 0m 0.37s 380441250 real 0m 0.46s 472342917 real 0m 0.55s 565745000 real 0m 0.64s 653705000 real 0m 0.73s 745983073 real 0m 0.82s 835870937 real 0m 0.93s 938043489 Using the max is not a conservative HP but after a 2 double check with the use of time, it can be accepted. for i in $(seq 1 100); do date +%s%N; usleep 10000; done | time cat >/dev/null real 0m 1.48s In fact, adding a little break between to date execution the average raised-up to 4800 us data=$(for i in $(seq 1 100); do a=$(date +%s%N); usleep 10000; b=$(date +%s%N); echo $((b-a)); usleep 10000; done | sort -n); echo "$data" | head -n1; echo "$data" | tail -n1; let avg=$(echo "$data" | tr '\n' '+')0; echo $(( avg / 100 )) 14482083 15539479 14757910 in both cases, 4758 us to be precise, which is still smaller than 5255. The most interesting thing is that time date might not be 100% reliable because both can access to RTC to take the times and therefore some kind of correlation / interference can be established. When a direct measure is not reliable, an indirect way to estimate a value can be searching for a value that best fits a regression relationship about cause-effect events. In this case 5255 can be a candidate for such indirect measure. The best candidate can be searched within [4500, 5600] interval. For each sequence 1..10 can be found the best fit. With 100 sequences 1..10, we will have 100 best fit and we can do statistics and regression with those numbers. We do not need that, obviously. Because the difference in performance and O(something) is wide enough that a conservative or a raw estimation is enough to see it. From bb at npl.de Sat Sep 9 15:14:51 2023 From: bb at npl.de (Bastian Bittorf) Date: Sat, 9 Sep 2023 15:14:51 +0000 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> Message-ID: <20230909151451.pjauhd3htpoc56d7@email> I tried to replicate your findings, but my slow embedded OpenWRT system (with musl libc) is so fast, that your tests always produce 0 seconds execution time. You are talking about nofork and cpu affinity: Can you repoduce your findings with booting OpenWRT on your board? bye, Bastian From roberto.foglietta at gmail.com Sun Sep 10 09:41:01 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 10 Sep 2023 11:41:01 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: <20230909151451.pjauhd3htpoc56d7@email> References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> <20230909151451.pjauhd3htpoc56d7@email> Message-ID: On Sat, 9 Sept 2023 at 17:14, Bastian Bittorf wrote: > > I tried to replicate your findings, but my slow embedded OpenWRT > system (with musl libc) is so fast, that your tests always produce > 0 seconds execution time. Shows the test shell code and the results. redfishos:/rootfs # time date +%s%N 1689574011301750356 real 0m 0.00s for i in $(seq 1 100); do date +%s%N; done | time cat >/dev/null real 0m 0.45s As you can see here above, time or date without the nanosecond support cannot have the proper time resolution to see the difference on a single run. Did you consider this? Best regards, R- From roberto.foglietta at gmail.com Sun Sep 10 09:51:36 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 10 Sep 2023 11:51:36 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> <20230909151451.pjauhd3htpoc56d7@email> Message-ID: On Sun, 10 Sept 2023 at 11:41, Roberto A. Foglietta wrote: > > On Sat, 9 Sept 2023 at 17:14, Bastian Bittorf wrote: > > > > I tried to replicate your findings, but my slow embedded OpenWRT > > system (with musl libc) is so fast, that your tests always produce > > 0 seconds execution time. > > Shows the test shell code and the results. > > redfishos:/rootfs # time date +%s%N > 1689574011301750356 > real 0m 0.00s > > for i in $(seq 1 100); do date +%s%N; done | time cat >/dev/null > real 0m 0.45s > > As you can see here above, time or date without the nanosecond > support cannot have the proper time resolution to see the difference > on a single run. Did you consider this? https://unix.stackexchange.com/questions/167968/date-in-milliseconds-on-openwrt-on-arduino-yun This link might help you to understand and address what I am speaking about (sub-seconds time resolution) I hope this helps, R- From roberto.foglietta at gmail.com Sun Sep 10 13:07:11 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 10 Sep 2023 15:07:11 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> <20230909151451.pjauhd3htpoc56d7@email> Message-ID: On Sun, 10 Sept 2023 at 11:51, Roberto A. Foglietta wrote: > > On Sun, 10 Sept 2023 at 11:41, Roberto A. Foglietta > wrote: > > > > On Sat, 9 Sept 2023 at 17:14, Bastian Bittorf wrote: > > > > > > I tried to replicate your findings, but my slow embedded OpenWRT > > > system (with musl libc) is so fast, that your tests always produce > > > 0 seconds execution time. > > > > Shows the test shell code and the results. > > > > redfishos:/rootfs # time date +%s%N > > 1689574011301750356 > > real 0m 0.00s > > > > for i in $(seq 1 100); do date +%s%N; done | time cat >/dev/null > > real 0m 0.45s > > > > As you can see here above, time or date without the nanosecond > > support cannot have the proper time resolution to see the difference > > on a single run. Did you consider this? > > https://unix.stackexchange.com/questions/167968/date-in-milliseconds-on-openwrt-on-arduino-yun > > This link might help you to understand and address what I am speaking > about (sub-seconds time resolution) > Notice that the O(1, grep) vs O(N, pidof) is always present but to see a big difference N >> 1, like 8 or 10 for example. Using 2 for seeing the difference also works, but the difference can be more easily confused with some other source of lantencies. However, the O(1) vs O(N) is a fact that affects every system that relies on busybox pidof. Instead, to clog the system it is required two more conditions: 1. taskset -pc 0 $$ 2. busybox compiled with NOFORK If one of these conditions is missing, then the system works fine within the O(1) vs O(N) limit. To slow down the system even more, a third condition can be used: 3. local var=$(command); echo "$var" Again, also this condition does not affect the system in a sensitive way if one of the two previous conditions is missing. This explains because few people might have noticed something weird in their system before. Those who have a single core/thread CPU and tried NOFORK for improving the script's timing, probably decided to go back to the previous configuration without further investigation, accepting for explanation that their system is quite limited compared with the nowadays standards. Those who play with taskset, usually use it for a specific binary or a set of binaries which are supposed to deliver a real-time service and therefore receive a single CPU for their own only. Both of these cases are not common plus most of the people do not care about script speed execution because they accept that shell scripting is something slow by its own nature. I consolidated this opinion after having seen a lot of scripts in the real-world. In fact, most of them can be easily improved by many times in speed. The last one I reviewed, and modified just to adapt to my system got boosted by 7x times. From dario.binacchi at amarulasolutions.com Sun Sep 10 15:01:54 2023 From: dario.binacchi at amarulasolutions.com (Dario Binacchi) Date: Sun, 10 Sep 2023 17:01:54 +0200 Subject: [RESEND, RFC PATCH] ip link: support for the CAN netlink Message-ID: <20230910150154.3428533-1-dario.binacchi@amarulasolutions.com> I developed this application to test the Linux kernel series [1]. As described in the cover letter I could not use the iproute2 package since the microcontroller is without MMU. cc: Marc Kleine-Budde [1] https://marc.info/?l=linux-netdev&m=167999323611710&w=2 Signed-off-by: Dario Binacchi --- configs/TEST_nommu_defconfig | 1 + networking/ip.c | 84 ++++++++++ networking/libiproute/iplink.c | 298 ++++++++++++++++++++++++++++++++- 3 files changed, 374 insertions(+), 9 deletions(-) diff --git a/configs/TEST_nommu_defconfig b/configs/TEST_nommu_defconfig index 415f5a8027f9..fa3e9632622a 100644 --- a/configs/TEST_nommu_defconfig +++ b/configs/TEST_nommu_defconfig @@ -703,6 +703,7 @@ CONFIG_FEATURE_INETD_RPC=y CONFIG_IP=y CONFIG_FEATURE_IP_ADDRESS=y CONFIG_FEATURE_IP_LINK=y +CONFIG_FEATURE_IP_LINK_CAN=y CONFIG_FEATURE_IP_ROUTE=y CONFIG_FEATURE_IP_TUNNEL=y CONFIG_FEATURE_IP_RULE=y diff --git a/networking/ip.c b/networking/ip.c index 7c320869958a..4959e9f25288 100644 --- a/networking/ip.c +++ b/networking/ip.c @@ -32,6 +32,14 @@ //config: help //config: Short form of "ip link" //config: +//config:config IPLINK_CAN +//config: bool "iplink for CAN (4.6 kb)" +//config: default n +//config: depends on IPLINK +//config: select FEATURE_IP_LINK_CAN +//config: help +//config: Short form of "ip link" for CAN +//config: //config:config IPROUTE //config: bool "iproute (15 kb)" //config: default y @@ -74,6 +82,13 @@ //config: help //config: Configure network devices with "ip". //config: +//config:config FEATURE_IP_LINK_CAN +//config: bool "ip link can" +//config: default n +//config: depends on IP_LINK_CAN +//config: help +//config: Configure CAN devices with "ip". +//config: //config:config FEATURE_IP_ROUTE //config: bool "ip route" //config: default y @@ -122,6 +137,7 @@ //applet:IF_IP( APPLET_NOEXEC(ip , ip , BB_DIR_SBIN, BB_SUID_DROP, ip )) //applet:IF_IPADDR( APPLET_NOEXEC(ipaddr , ipaddr , BB_DIR_SBIN, BB_SUID_DROP, ipaddr )) //applet:IF_IPLINK( APPLET_NOEXEC(iplink , iplink , BB_DIR_SBIN, BB_SUID_DROP, iplink )) +//applet:IF_IPLINK_CAN(APPLET_NOEXEC(iplinkcan , iplinkcan , BB_DIR_SBIN, BB_SUID_DROP, iplinkcan)) //applet:IF_IPROUTE( APPLET_NOEXEC(iproute , iproute , BB_DIR_SBIN, BB_SUID_DROP, iproute )) //applet:IF_IPRULE( APPLET_NOEXEC(iprule , iprule , BB_DIR_SBIN, BB_SUID_DROP, iprule )) //applet:IF_IPTUNNEL(APPLET_NOEXEC(iptunnel, iptunnel, BB_DIR_SBIN, BB_SUID_DROP, iptunnel)) @@ -130,6 +146,7 @@ //kbuild:lib-$(CONFIG_IP) += ip.o //kbuild:lib-$(CONFIG_IPADDR) += ip.o //kbuild:lib-$(CONFIG_IPLINK) += ip.o +//kbuild:lib-$(CONFIG_IPLINK_CAN) += ip.o //kbuild:lib-$(CONFIG_IPROUTE) += ip.o //kbuild:lib-$(CONFIG_IPRULE) += ip.o //kbuild:lib-$(CONFIG_IPTUNNEL) += ip.o @@ -149,10 +166,16 @@ //usage: "ipaddr show|flush [dev IFACE] [scope SCOPE] [to PREFIX] [label PATTERN]" //usage: //--------------123456789.123456789.123456789.123456789.123456789.123456789.123456789.123....79 +//usage:#if ENABLE_FEATURE_IP_LINK_CAN +//usage:#define iplink_type_usage "\n [type TYPE ARGS]" +//usage:#else +//usage:#define iplink_type_usage "" +//usage:#endif //usage:#define iplink_trivial_usage //usage: /*Usage:iplink*/"set IFACE [up|down] [arp on|off] [multicast on|off]\n" //usage: " [promisc on|off] [mtu NUM] [name NAME] [qlen NUM] [address MAC]\n" //usage: " [master IFACE | nomaster] [netns PID]" +//usage: IF_FEATURE_IP_LINK(iplink_type_usage) // * short help shows only "set" command, long help continues (with just one "\n") // * and shows all other commands: //usage:#define iplink_full_usage "\n" @@ -207,6 +230,59 @@ // bond_slave | ipvlan | geneve | bridge_slave | vrf } //usage: //--------------123456789.123456789.123456789.123456789.123456789.123456789.123456789.123....79 +//usage:#define iplinkcan_trivial_usage +//usage: /*Usage:iplinkcan*/"set DEVICE type can" +//usage:#define iplinkcan_full_usage "\n\n" +//usage: " [bitrate BITRATE [sample-point SAMPLE-POINT]] |\n" +//usage: " [tq TQ prop-seg PROP_SEG phase-seg1 PHASE-SEG1\n" +//usage: " phase-seg2 PHASE-SEG2 [sjw SJW]]\n" +//usage: "\n" +//usage: " [dbitrate BITRATE [dsample-point SAMPLE-POINT]] |\n" +//usage: " [dtq TQ dprop-seg PROP_SEG dphase-seg1 PHASE-SEG1\n" +//usage: " dphase-seg2 PHASE-SEG2 [dsjw SJW]]\n" +//usage: "\n" +//usage: " [loopback on|off] [listen-only on|off] [triple-sampling on|off]\n" +//usage: " [one-shot on|off] [berr-reporting on|off]\n" +//usage: " [fd on|off] [fd-non-iso on|off] [presume-ack on|off]\n" +//usage: "\n" +//usage: " [restart-ms TIME-MS] [restart]\n" +//usage: "\n" +//usage: " [termination 0..65535]\n" +//usage: +//upstream man ip-link-can: +//Usage: ip link set DEVICE type can +// [ bitrate BITRATE [ sample-point SAMPLE-POINT] ] | +// [ tq TQ prop-seg PROP_SEG phase-seg1 PHASE-SEG1 +// phase-seg2 PHASE-SEG2 [ sjw SJW ] ] +// +// [ dbitrate BITRATE [ dsample-point SAMPLE-POINT] ] | +// [ dtq TQ dprop-seg PROP_SEG dphase-seg1 PHASE-SEG1 +// dphase-seg2 PHASE-SEG2 [ dsjw SJW ] ] +// +// [ loopback { on | off } ] +// [ listen-only { on | off } ] +// [ triple-sampling { on | off } ] +// [ one-shot { on | off } ] +// [ berr-reporting { on | off } ] +// [ fd { on | off } ] +// [ fd-non-iso { on | off } ] +// [ presume-ack { on | off } ] +// +// [ restart-ms TIME-MS ] +// [ restart ] +// +// [ termination { 0..65535 } ] +// +// Where: BITRATE := { 1..1000000 } +// SAMPLE-POINT := { 0.000..0.999 } +// TQ := { NUMBER } +// PROP-SEG := { 1..8 } +// PHASE-SEG1 := { 1..8 } +// PHASE-SEG2 := { 1..8 } +// SJW := { 1..4 } +// RESTART-MS := { 0 | NUMBER } +//usage: +//--------------123456789.123456789.123456789.123456789.123456789.123456789.123456789.123....79 //usage:#define iproute_trivial_usage //usage: "list|flush|add|del|change|append|replace|test ROUTE" //usage:#define iproute_full_usage "\n\n" @@ -327,6 +403,7 @@ typedef int FAST_FUNC (*ip_func_ptr_t)(char**); #if ENABLE_IPADDR \ || ENABLE_IPLINK \ + || ENABLE_IPLINK_CAN \ || ENABLE_IPROUTE \ || ENABLE_IPRULE \ || ENABLE_IPTUNNEL \ @@ -352,6 +429,13 @@ int iplink_main(int argc UNUSED_PARAM, char **argv) return ip_do(do_iplink, argv); } #endif +#if ENABLE_IPLINK_CAN +int iplinkcan_main(int argc, char **argv) MAIN_EXTERNALLY_VISIBLE; +int iplinkcan_main(int argc UNUSED_PARAM, char **argv) +{ + return ip_do(do_iplink, argv); +} +#endif #if ENABLE_IPROUTE int iproute_main(int argc, char **argv) MAIN_EXTERNALLY_VISIBLE; int iproute_main(int argc UNUSED_PARAM, char **argv) diff --git a/networking/libiproute/iplink.c b/networking/libiproute/iplink.c index 68d1990445fe..f97169714d0f 100644 --- a/networking/libiproute/iplink.c +++ b/networking/libiproute/iplink.c @@ -11,10 +11,17 @@ #include #include +#include #include "ip_common.h" /* #include "libbb.h" is inside */ #include "rt_names.h" #include "utils.h" +#if ENABLE_FEATURE_IP_LINK_CAN +#define ENABLE_FEATURE_IP_LINK_IFACE 1 +#else +#define ENABLE_FEATURE_IP_LINK_IFACE 0 +#endif + #undef ETH_P_8021AD #define ETH_P_8021AD 0x88A8 #undef VLAN_FLAG_REORDER_HDR @@ -28,6 +35,11 @@ #undef IFLA_VLAN_PROTOCOL #define IFLA_VLAN_PROTOCOL 5 +#ifndef NLMSG_TAIL +#define NLMSG_TAIL(nmsg) \ + ((struct rtattr *) (((void *) (nmsg)) + NLMSG_ALIGN((nmsg)->nlmsg_len))) +#endif + #ifndef IFLA_LINKINFO # define IFLA_LINKINFO 18 # define IFLA_INFO_KIND 1 @@ -55,6 +67,13 @@ struct ifla_vlan_flags { #define str_on_off "on\0""off\0" +enum { + PARM_on = 0, + PARM_off +}; + +typedef void FAST_FUNC(*ip_type_set_func_ptr_t)(char*, char**); + /* Exits on error */ static int get_ctl_fd(void) { @@ -241,10 +260,261 @@ static void die_must_be_on_off(const char *msg) bb_error_msg_and_die("argument of \"%s\" must be \"on\" or \"off\"", msg); } +#if ENABLE_FEATURE_IP_LINK_CAN +static float FAST_FUNC get_float(char *arg, const char *errmsg) +{ + float ret; + char *ptr; + + if (!arg || !*arg) + invarg_1_to_2(arg, errmsg); /* does not return */ + + ret = strtof(arg, &ptr); + if (!ptr || ptr == arg || *ptr) + invarg_1_to_2(arg, errmsg); /* does not return */ + + return ret; +} + +static void do_set_can(char *dev, char **argv) +{ + struct can_bittiming bt = {}, dbt = {}; + struct can_ctrlmode cm = {}; + char *keyword; + static const char keywords[] ALIGN1 = + "bitrate\0""sample-point\0""tq\0" + "prop-seg\0""phase-seg1\0""phase-seg2\0""sjw\0" + "dbitrate\0""dsample-point\0""dtq\0" + "dprop-seg\0""dphase-seg1\0""dphase-seg2\0""dsjw\0" + "loopback\0""listen-only\0""triple-sampling\0" + "one-shot\0""berr-reporting\0" + "fd\0""fd-non-iso\0""presume-ack\0" + "cc-len8-dlc\0""restart\0""restart-ms\0" + "termination\0"; + enum { ARG_bitrate = 0, ARG_sample_point, ARG_tq, + ARG_prop_seg, ARG_phase_seg1, ARG_phase_seg2, ARG_sjw, + ARG_dbitrate, ARG_dsample_point, ARG_dtq, + ARG_dprop_seg, ARG_dphase_seg1, ARG_dphase_seg2, ARG_dsjw, + ARG_loopback, ARG_listen_only, ARG_triple_sampling, + ARG_one_shot, ARG_berr_reporting, + ARG_fd, ARG_fd_non_iso, ARG_presume_ack, + ARG_cc_len8_dlc, ARG_restart, ARG_restart_ms, + ARG_termination }; + struct rtnl_handle rth; + struct { + struct nlmsghdr n; + struct ifinfomsg i; + char buf[1024]; + } req; + size_t dev_len; + struct rtattr *linkinfo, *data; + smalluint key, param; + + memset(&req, 0, sizeof(req)); + + req.n.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)); + req.n.nlmsg_flags = NLM_F_REQUEST; + req.n.nlmsg_type = RTM_NEWLINK; + req.i.ifi_family = preferred_family; + xrtnl_open(&rth); + req.i.ifi_index = xll_name_to_index(dev); + dev_len = strlen(dev); + if (dev_len < 2 || dev_len > IFNAMSIZ) + invarg_1_to_2(dev, "dev"); + + addattr_l(&req.n, sizeof(req), IFLA_IFNAME, dev, dev_len); + linkinfo = NLMSG_TAIL(&req.n); + addattr_l(&req.n, sizeof(req), IFLA_LINKINFO, NULL, 0); + addattr_l(&req.n, sizeof(req), IFLA_INFO_KIND, (void *)"can", + strlen("can")); + data = NLMSG_TAIL(&req.n); + addattr_l(&req.n, sizeof(req), IFLA_INFO_DATA, NULL, 0); + while (*argv) { + key = index_in_substrings(keywords, *argv); + keyword = *argv; + //printf("%s: key: %d, *argv: %s\n", __func__, key, *argv); + switch (key) { + case ARG_bitrate: + case ARG_tq: + case ARG_prop_seg: + case ARG_phase_seg1: + case ARG_phase_seg2: + case ARG_sjw: { + __u32 *val; + + NEXT_ARG(); + if (key == ARG_bitrate) + val = &bt.bitrate; + else if (key == ARG_tq) + val = &bt.tq; + else if (key == ARG_prop_seg) + val = &bt.prop_seg; + else if (key == ARG_phase_seg1) + val = &bt.phase_seg1; + else if (key == ARG_phase_seg2) + val = &bt.phase_seg2; + else + val = &bt.sjw; + + *val = get_u32(*argv, keyword); + break; + } + case ARG_sample_point: { + float sp; + + NEXT_ARG(); + sp = get_float(*argv, keyword); + bt.sample_point = (__u32)(sp * 1000); + break; + } + case ARG_dbitrate: + case ARG_dtq: + case ARG_dprop_seg: + case ARG_dphase_seg1: + case ARG_dphase_seg2: + case ARG_dsjw: { + __u32 *val; + + NEXT_ARG(); + if (key == ARG_dbitrate) + val = &dbt.bitrate; + else if (key == ARG_dtq) + val = &dbt.tq; + else if (key == ARG_dprop_seg) + val = &dbt.prop_seg; + else if (key == ARG_dphase_seg1) + val = &dbt.phase_seg1; + else if (key == ARG_dphase_seg2) + val = &dbt.phase_seg2; + else + val = &dbt.sjw; + + *val = get_u32(*argv, keyword); + break; + } + case ARG_dsample_point: { + float sp; + + NEXT_ARG(); + sp = get_float(*argv, keyword); + dbt.sample_point = (__u32)(sp * 1000); + break; + } + case ARG_loopback: + case ARG_listen_only: + case ARG_triple_sampling: + case ARG_one_shot: + case ARG_berr_reporting: + case ARG_fd: + case ARG_fd_non_iso: + case ARG_presume_ack: + case ARG_cc_len8_dlc: { + __u32 flag = 0; + + NEXT_ARG(); + param = index_in_strings(str_on_off, *argv); + if (param < 0) + die_must_be_on_off(keyword); + + if (key == ARG_loopback) + flag = CAN_CTRLMODE_LOOPBACK; + else if (key == ARG_listen_only) + flag = CAN_CTRLMODE_LISTENONLY; + else if (key == ARG_triple_sampling) + flag = CAN_CTRLMODE_3_SAMPLES; + else if (key == ARG_one_shot) + flag = CAN_CTRLMODE_ONE_SHOT; + else if (key == ARG_berr_reporting) + flag = CAN_CTRLMODE_BERR_REPORTING; + else if (key == ARG_fd) + flag = CAN_CTRLMODE_FD; + else if (key == ARG_fd_non_iso) + flag = CAN_CTRLMODE_FD_NON_ISO; + else if (key == ARG_presume_ack) + flag = CAN_CTRLMODE_PRESUME_ACK; + else +#if defined(CAN_CTRLMODE_CC_LEN8_DLC) + flag = CAN_CTRLMODE_CC_LEN8_DLC; +#else + die_must_be_on_off(keyword); +#endif + cm.mask |= flag; + if (param == PARM_on) + cm.flags |= flag; + + break; + } + case ARG_restart: { + __u32 val = 1; + + NEXT_ARG(); + addattr_l(&req.n, sizeof(req), IFLA_CAN_RESTART, &val, sizeof(val)); + break; + } + case ARG_restart_ms: { + __u32 val; + + NEXT_ARG(); + val = get_u32(*argv, keyword); + addattr_l(&req.n, sizeof(req), IFLA_CAN_RESTART_MS, &val, sizeof(val)); + break; + } + case ARG_termination: { + __u16 val; + + NEXT_ARG(); + val = get_u16(*argv, keyword); + addattr_l(&req.n, sizeof(req), IFLA_CAN_TERMINATION, &val, sizeof(val)); + break; + } + default: + break; + } + + argv++; + } + + if (bt.bitrate || bt.tq) + addattr_l(&req.n, sizeof(req), IFLA_CAN_BITTIMING, &bt, sizeof(bt)); + + if (cm.mask) + addattr_l(&req.n, sizeof(req), IFLA_CAN_CTRLMODE, &cm, sizeof(cm)); + + data->rta_len = (void *)NLMSG_TAIL(&req.n) - (void *)data; + linkinfo->rta_len = (void *)NLMSG_TAIL(&req.n) - (void *)linkinfo; + + if (rtnl_talk(&rth, &req.n, 0, 0, NULL, NULL, NULL) < 0) + xfunc_die(); +} +#endif + +#if ENABLE_FEATURE_IP_LINK_IFACE +static void do_set_iface(char *type, char *dev, char **argv) +{ + static const char keywords[] ALIGN1 = "" + IF_FEATURE_IP_LINK_CAN("can\0") + ; + static const ip_type_set_func_ptr_t funcs[] ALIGN_PTR = { + IF_FEATURE_IP_LINK_CAN(do_set_can,) + }; + ip_type_set_func_ptr_t func; + int key; + + key = index_in_substrings(keywords, type); + if (key < 0) + return; + func = funcs[key]; + func(dev, argv); +} +#endif + /* Return value becomes exitcode. It's okay to not return at all */ static int do_set(char **argv) { char *dev = NULL; +#if ENABLE_FEATURE_IP_LINK_IFACE + char *type = NULL; +#endif uint32_t mask = 0; uint32_t flags = 0; int qlen = -1; @@ -261,18 +531,24 @@ static int do_set(char **argv) "up\0""down\0""name\0""mtu\0""qlen\0""multicast\0" "arp\0""promisc\0""address\0""netns\0" "master\0""nomaster\0" +#if ENABLE_FEATURE_IP_LINK_IFACE + "type\0" +#endif "dev\0" /* must be last */; enum { ARG_up = 0, ARG_down, ARG_name, ARG_mtu, ARG_qlen, ARG_multicast, ARG_arp, ARG_promisc, ARG_addr, ARG_netns, ARG_master, ARG_nomaster, +#if ENABLE_FEATURE_IP_LINK_IFACE + ARG_type, +#endif ARG_dev }; - enum { PARM_on = 0, PARM_off }; smalluint key; while (*argv) { /* substring search ensures that e.g. "addr" and "address" * are both accepted */ key = index_in_substrings(keywords, *argv); + //printf("%s: key: %d, *argv: %s\n", __func__, key, *argv); if (key == ARG_up) { mask |= IFF_UP; flags |= IFF_UP; @@ -304,6 +580,13 @@ static int do_set(char **argv) } else if (key == ARG_netns) { NEXT_ARG(); netns = get_unsigned(*argv, "netns"); +#if ENABLE_FEATURE_IP_LINK_IFACE + } else if (key == ARG_type) { + NEXT_ARG(); + type = *argv; + argv++; + break; +#endif } else if (key >= ARG_dev) { /* ^^^^^^ ">=" here results in "dev IFACE" treated as default */ if (key == ARG_dev) { @@ -311,6 +594,7 @@ static int do_set(char **argv) } if (dev) duparg2("dev", *argv); + dev = *argv; } else { /* "on|off" options */ @@ -496,6 +780,10 @@ static int do_set(char **argv) } if (mask) do_chflags(dev, flags, mask); +#if ENABLE_FEATURE_IP_LINK_IFACE + if (type) + do_set_iface(type, dev, argv); +#endif return 0; } @@ -531,10 +819,6 @@ static void vlan_parse_opt(char **argv, struct nlmsghdr *n, unsigned int size) PROTO_8021Q = 0, PROTO_8021AD, }; - enum { - PARM_on = 0, - PARM_off - }; int arg; uint16_t id, proto; struct ifla_vlan_flags flags = {}; @@ -610,10 +894,6 @@ static void vrf_parse_opt(char **argv, struct nlmsghdr *n, unsigned int size) addattr_l(n, size, IFLA_VRF_TABLE, &table, sizeof(table)); } -#ifndef NLMSG_TAIL -#define NLMSG_TAIL(nmsg) \ - ((struct rtattr *) (((void *) (nmsg)) + NLMSG_ALIGN((nmsg)->nlmsg_len))) -#endif /* Return value becomes exitcode. It's okay to not return at all */ static int do_add_or_delete(char **argv, const unsigned rtm) { -- 2.32.0 From roberto.foglietta at gmail.com Sun Sep 10 17:12:01 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 10 Sep 2023 19:12:01 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> <20230909151451.pjauhd3htpoc56d7@email> Message-ID: On Sun, 10 Sept 2023 at 15:07, Roberto A. Foglietta wrote: > Notice that the O(1, grep) vs O(N, pidof) is always present but to see > a big difference N >> 1, like 8 or 10 for example. Using 2 for seeing > the difference also works, but the difference can be more easily > confused with some other source of lantencies. However, the O(1) vs > O(N) is a fact that affects every system that relies on busybox pidof. > This is O(N) with N the number of parameters, 100 just to be sure that time can do its job with 1/100s pidof $(seq 1000 1100) | time cat real 0m 9.57s pidof $(seq 1000 1050) | time cat real 0m 4.80s pidof 1000 | time cat real 0m 0.09s This is O(1) clearly comw_pid $(seq 1000 1100) | time cat real 0m 0.12s The system is running at 300MHz fixed frequency with 4 CPUs enabled and 4 CPUs off-line. taskset -pc $$ pid 1100's current affinity list: 0-3 We may wonder who is going to call pidof with 100 arguments but this is another story. From roberto.foglietta at gmail.com Mon Sep 11 15:28:00 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 11 Sep 2023 17:28:00 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: <20230909151451.pjauhd3htpoc56d7@email> References: <20230904165312.tct4vx4h67qasttz@email> <20230907074427.j23zmyp6rpzlitxv@email> <20230909151451.pjauhd3htpoc56d7@email> Message-ID: On Sat, 9 Sept 2023 at 17:14, Bastian Bittorf wrote: > > I tried to replicate your findings, but my slow embedded OpenWRT > system (with musl libc) is so fast, that your tests always produce > 0 seconds execution time. You are talking about nofork and cpu affinity: About the system speed, I took a comparison with my standard Ubuntu installation and the latency in accessing to /proc is a matter of fact. It depends on the kernel that every /proc access updates all the values including cpu frequency and especially for these values, it needs some time to determine the reliability of the value. This is what I understood reading documentation around, but I did not try anything particular to check/confute this. Using strace to check the timing of every operation, almost all the access to proc by busybox pidof have the same duration and no particular operation consumes a lot of time. This excludes that there is a single instruction/command that creates a bottle-neck. Therefore, I investigated the kernel debug system, deactivated it, etc. etc. and finally I took in consideration that in some "advanced" systems the /proc is slow to access while in bare-simple system it is not a problem (e.g. CPU frequency scaling does not exist at all). Aligning the max frequency with the minimum does not change anything about this aspect. If you have suggestions, you are welcome. Best regards, R- From zhuyan2015 at foxmail.com Mon Sep 11 16:03:09 2023 From: zhuyan2015 at foxmail.com (Yan Zhu) Date: Tue, 12 Sep 2023 00:03:09 +0800 Subject: [PATCH] docproc: avoid segfault during file closing Message-ID: In the function find_export_symbols, since the fopen file does not exit when it fails, there is a dereference problem in fclose(fp), which will cause a segmentation fault. Signed-off-by: Yan Zhu --- scripts/basic/docproc.c | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/basic/docproc.c b/scripts/basic/docproc.c index 4464e1874..8828901a1 100644 --- a/scripts/basic/docproc.c +++ b/scripts/basic/docproc.c @@ -180,6 +180,7 @@ void find_export_symbols(char * filename) { fprintf(stderr, "docproc: "); perror(real_filename); + exit(1); } while (fgets(line, MAXLINESZ, fp)) { unsigned char *p; -- 2.33.0 From ben.wolsieffer at hefring.com Mon Sep 11 20:43:15 2023 From: ben.wolsieffer at hefring.com (Ben Wolsieffer) Date: Mon, 11 Sep 2023 16:43:15 -0400 Subject: [RESEND PATCH] httpd: fix index.html with CGI and no basic auth Message-ID: <20230911204315.2138270-1-ben.wolsieffer@hefring.com> From: Ben Wolsieffer The URL gets truncated if ENABLE_FEATURE_HTTPD_BASIC_AUTH or ENABLE_FEATURE_HTTPD_CGI is set, but it is only restored if ENABLE_FEATURE_HTTPD_BASIC_AUTH is set. This breaks index.html redirects for configurations that have CGI enabled but basic auth disabled. Fixes: 91a58b207ea0 ("httpd: no need to strcpy() when we only need to copy one byte") Signed-off-by: Ben Wolsieffer --- networking/httpd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/networking/httpd.c b/networking/httpd.c index ddcb03bca..0098c7aea 100644 --- a/networking/httpd.c +++ b/networking/httpd.c @@ -2616,7 +2616,7 @@ static void handle_incoming_and_exit(const len_and_sockaddr *fromAddr) /* !CGI: it can be only GET or HEAD */ #endif -#if ENABLE_FEATURE_HTTPD_BASIC_AUTH +#if ENABLE_FEATURE_HTTPD_BASIC_AUTH || ENABLE_FEATURE_HTTPD_CGI /* Restore truncated .../index.html */ if (urlp[-1] == '/') urlp[0] = index_page[0]; -- 2.42.0 From roberto.foglietta at gmail.com Tue Sep 12 08:04:49 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 12 Sep 2023 10:04:49 +0200 Subject: pidof and pgrep are slower than a grep /proc/[1-9]*/comm In-Reply-To: References: <20230904165312.tct4vx4h67qasttz@email> Message-ID: On Wed, 6 Sept 2023 at 18:59, Roberto A. Foglietta wrote: > comw_pid() > { > local str > # RAF: the /proc/$pid/comm contains the first 15 chars + ending \n > str=$(echo -n $*| tr ' ' '\n'| sed -e "s,\(.\{15\}\).*,^\\1$,"| tr '\n' '|') > str=$(grep -nE "$str" /proc/[1-9]*/comm 2>/dev/null | cut -d/ -f3) > echo $str | grep --color=never . > } For those who cares, this is the new version of the function with the regexp correction: comw_pid() { local str # RAF: the /proc/$pid/comm contains the first 15 chars + an ending \n str=$(echo -n $*|tr ' ' '\n'|sed -e 's/\(.\{1,15\}\).*/^\1$/'| tr '\n' '|') str=$(grnc -nE "$str" /proc/[1-9]*/comm 2>/dev/null | cut -d/ -f3) echo $str | grnc . } It can be a replace of pgrep/pidof and can use regular expression like these: redfishos:~ # comw_pid yamui 623 redfishos:~ # comw_pid 'yamui.*' 619 623 703 redfishos:~ # comw_pid '.*sh' 13900 21934 21935 It is much more faster than using busybox pidof and here there is a practical example: redfishos:~ # comw_pid '[a-z]*sh' 'yamui.*' | time cat 13900 23006 23014 23015 619 623 703 real 0m 0.13s redfishos:~ # time pidof bash ash sh zsh csh dash yamui 13900 13899 11319 722 721 658 1 623 real 0m 0.62s Searching for every shell plus a process of your own, can be considered something real and not a corner case like querying 100 process names. Best regards, R- From roberto.foglietta at gmail.com Tue Sep 12 08:50:05 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 12 Sep 2023 10:50:05 +0200 Subject: busybox hdparm -F is not supported but can be very useful Message-ID: Hi, I found this option in hdparm which is not supported by busybox hdparm -F Flush the on-drive write cache buffer (older drives may not implement this). I did not verify the internals of hdparm -F but relying on the man hdparm, I think that it is an interesting feature to implement also in busybox hdparm. Especially for those device for which the shutdown can be controlled with external hardware or forced brutally but need to know that almost everything is saved up to that point. I hope this helps, R- From steffen at sdaoden.eu Tue Sep 12 21:19:25 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Tue, 12 Sep 2023 23:19:25 +0200 Subject: busybox hdparm -F is not supported but can be very useful In-Reply-To: References: Message-ID: <20230912211925.MzB0L%steffen@sdaoden.eu> Roberto A. Foglietta wrote in : |I hope this helps, R- i have this in a file .busybox-TODO but i think it will never happen: sysklogd (notify) unshare (reaper) acpid->2 rdate (-n, -a, [-v]) less -e s-s-d status --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From troglobit at gmail.com Wed Sep 13 08:37:27 2023 From: troglobit at gmail.com (Joachim Wiberg) Date: Wed, 13 Sep 2023 10:37:27 +0200 Subject: [PATCH 1/1] logger: add '-i' flag for optional PID to syslog messages Message-ID: <20230913083727.2321695-1-troglobit@gmail.com> The -i flag in the util-linux, and upstream sysklogd, projects add PID to syslog messages, using the LOG_PID flag. Signed-off-by: Joachim Wiberg --- sysklogd/logger.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sysklogd/logger.c b/sysklogd/logger.c index 04b2c8e3b..8424ae206 100644 --- a/sysklogd/logger.c +++ b/sysklogd/logger.c @@ -21,9 +21,10 @@ //kbuild:lib-$(CONFIG_LOGGER) += syslogd_and_logger.o //usage:#define logger_trivial_usage -//usage: "[-s] [-t TAG] [-p PRIO] [MESSAGE]" +//usage: "[-i] [-s] [-t TAG] [-p PRIO] [MESSAGE]" //usage:#define logger_full_usage "\n\n" //usage: "Write MESSAGE (or stdin) to syslog\n" +//usage: "\n -i Log PID of logger to system log" //usage: "\n -s Log to stderr as well as the system log" //usage: "\n -t TAG Log using the specified tag (defaults to user name)" //usage: "\n -p PRIO Priority (number or FACILITY.LEVEL pair)" @@ -105,10 +106,12 @@ int logger_main(int argc UNUSED_PARAM, char **argv) str_t = uid2uname_utoa(geteuid()); /* Parse any options */ - opt = getopt32(argv, "p:st:", &str_p, &str_t); + opt = getopt32(argv, "p:st:i", &str_p, &str_t); if (opt & 0x2) /* -s */ i |= LOG_PERROR; + if (opt & 0x8) /* -i */ + i |= LOG_PID; //if (opt & 0x4) /* -t */ openlog(str_t, i, 0); i = LOG_USER | LOG_NOTICE; -- 2.34.1 From troglobit at gmail.com Wed Sep 13 08:39:43 2023 From: troglobit at gmail.com (Joachim Wiberg) Date: Wed, 13 Sep 2023 10:39:43 +0200 Subject: [PATCH 1/1] login: add support for shadow passwords Message-ID: <20230913083943.2322205-1-troglobit@gmail.com> login, on fallback from PAM, or when PAM support is not enabled, checks pw->pw_passwd for locked ("!") or passwordless ("*") accounts. However, on systems with shadow passwords the first character will always be "x". This patch adds shadow password support from the passwd tool, letting the user end up in "Login incorrect" rather than the "login: bad salt" case, which could be used by an attacker to guess the state of accounts. Signed-off-by: Joachim Wiberg --- loginutils/login.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/loginutils/login.c b/loginutils/login.c index b02be2176..0e7f20844 100644 --- a/loginutils/login.c +++ b/loginutils/login.c @@ -345,6 +345,11 @@ int login_main(int argc UNUSED_PARAM, char **argv) #endif #if ENABLE_LOGIN_SESSION_AS_CHILD pid_t child_pid; +#endif +#if ENABLE_FEATURE_SHADOWPASSWDS + /* Using _r function to avoid pulling in static buffers */ + struct spwd spw, *result = NULL; + char buffer[256]; #endif IF_FEATURE_UTMP(pid_t my_pid;) @@ -493,6 +498,16 @@ int login_main(int argc UNUSED_PARAM, char **argv) goto fake_it; } +#if ENABLE_FEATURE_SHADOWPASSWDS + if (getspnam_r(pw->pw_name, &spw, buffer, sizeof(buffer), &result) + || !result || strcmp(result->sp_namp, pw->pw_name)) { + strcpy(username, "UNKNOWN"); + goto fake_it; + } else { + pw->pw_passwd = result->sp_pwdp; + } +#endif + if (pw->pw_passwd[0] == '!' || pw->pw_passwd[0] == '*') goto auth_failed; -- 2.34.1 From troglobit at gmail.com Wed Sep 13 08:41:37 2023 From: troglobit at gmail.com (Joachim Wiberg) Date: Wed, 13 Sep 2023 10:41:37 +0200 Subject: [PATCH 1/1] adduser: clarify adduser -D behavior and add -d for SSH key login Message-ID: <20230913084137.2322626-1-troglobit@gmail.com> Clarify that -D locks the account (!), then add -d to create an account for which password login is disabled (*) but the user can log in with SSH keys. This also adjusts the long option --disabled-password, which was mapped to -D, probably mistakenly. With this change BusyBox adduser behaves the same as Debian's --disabled-login and --disabled-password. Fixes #10981 Signed-off-by: Joachim Wiberg --- loginutils/adduser.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/loginutils/adduser.c b/loginutils/adduser.c index d3c795afa..cf6a0264a 100644 --- a/loginutils/adduser.c +++ b/loginutils/adduser.c @@ -62,7 +62,8 @@ //usage: "\n -s SHELL Login shell" //usage: "\n -G GRP Group" //usage: "\n -S Create a system user" -//usage: "\n -D Don't assign a password" +//usage: "\n -D Don't assign a password (locked account)" +//usage: "\n -d Like -D but allow login using SSH keys" //usage: "\n -H Don't create home directory" //usage: "\n -u UID User id" //usage: "\n -k SKEL Skeleton directory (/etc/skel)" @@ -82,10 +83,11 @@ #define OPT_SHELL (1 << 2) #define OPT_GID (1 << 3) #define OPT_DONT_SET_PASS (1 << 4) -#define OPT_SYSTEM_ACCOUNT (1 << 5) -#define OPT_DONT_MAKE_HOME (1 << 6) -#define OPT_UID (1 << 7) -#define OPT_SKEL (1 << 8) +#define OPT_DISABLED_PASS (1 << 5) +#define OPT_SYSTEM_ACCOUNT (1 << 6) +#define OPT_DONT_MAKE_HOME (1 << 7) +#define OPT_UID (1 << 8) +#define OPT_SKEL (1 << 9) /* remix */ /* recoded such that the uid may be passed in *p */ @@ -168,7 +170,8 @@ static const char adduser_longopts[] ALIGN1 = "gecos\0" Required_argument "g" "shell\0" Required_argument "s" "ingroup\0" Required_argument "G" - "disabled-password\0" No_argument "D" + "disabled-password\0" No_argument "d" + "disabled-login\0" No_argument "D" "empty-password\0" No_argument "D" "system\0" No_argument "S" "no-create-home\0" No_argument "H" @@ -202,10 +205,10 @@ int adduser_main(int argc UNUSED_PARAM, char **argv) pw.pw_dir = NULL; opts = getopt32long(argv, "^" - "h:g:s:G:DSHu:k:" + "h:g:s:G:DdSHu:k:" /* at least one and at most two non-option args */ /* disable interactive passwd for system accounts */ - "\0" "-1:?2:SD", + "\0" "-1:?2:SDd", adduser_longopts, &pw.pw_dir, &pw.pw_gecos, &pw.pw_shell, &usegroup, &uid, &skel @@ -263,7 +266,8 @@ int adduser_main(int argc UNUSED_PARAM, char **argv) * 8. unix date when login expires (i.e. when it may no longer be used) */ /* fields: 2 3 4 5 6 78 */ - p = xasprintf("!:%u:0:99999:7:::", (unsigned)(time(NULL)) / (24*60*60)); + p = xasprintf("%c:%u:0:99999:7:::", (opts & OPT_DISABLED_PASS) ? '*' : '!', + (unsigned)(time(NULL)) / (24*60*60)); /* ignore errors: if file is missing we suppose admin doesn't want it */ update_passwd(bb_path_shadow_file, pw.pw_name, p, NULL); if (ENABLE_FEATURE_CLEAN_UP) @@ -305,7 +309,7 @@ int adduser_main(int argc UNUSED_PARAM, char **argv) } } - if (!(opts & OPT_DONT_SET_PASS)) { + if (!(opts & (OPT_DONT_SET_PASS | OPT_DISABLED_PASS))) { /* interactively set passwd */ passwd_wrapper(pw.pw_name); } -- 2.34.1 From farmatito at tiscali.it Wed Sep 13 21:08:01 2023 From: farmatito at tiscali.it (tito) Date: Wed, 13 Sep 2023 23:08:01 +0200 Subject: [PATCH 1/1] login: add support for shadow passwords In-Reply-To: <20230913083943.2322205-1-troglobit@gmail.com> References: <20230913083943.2322205-1-troglobit@gmail.com> Message-ID: <20230913230801.4e763157@devuan> On Wed, 13 Sep 2023 10:39:43 +0200 Joachim Wiberg wrote: > login, on fallback from PAM, or when PAM support is not enabled, checks > pw->pw_passwd for locked ("!") or passwordless ("*") accounts. However, > on systems with shadow passwords the first character will always be "x". > > This patch adds shadow password support from the passwd tool, letting > the user end up in "Login incorrect" rather than the "login: bad salt" > case, which could be used by an attacker to guess the state of accounts. > > Signed-off-by: Joachim Wiberg > --- > loginutils/login.c | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/loginutils/login.c b/loginutils/login.c > index b02be2176..0e7f20844 100644 > --- a/loginutils/login.c > +++ b/loginutils/login.c > @@ -345,6 +345,11 @@ int login_main(int argc UNUSED_PARAM, char **argv) > #endif > #if ENABLE_LOGIN_SESSION_AS_CHILD > pid_t child_pid; > +#endif > +#if ENABLE_FEATURE_SHADOWPASSWDS > + /* Using _r function to avoid pulling in static buffers */ > + struct spwd spw, *result = NULL; > + char buffer[256]; > #endif > IF_FEATURE_UTMP(pid_t my_pid;) > > @@ -493,6 +498,16 @@ int login_main(int argc UNUSED_PARAM, char **argv) > goto fake_it; > } > > +#if ENABLE_FEATURE_SHADOWPASSWDS > + if (getspnam_r(pw->pw_name, &spw, buffer, sizeof(buffer), &result) > + || !result || strcmp(result->sp_namp, pw->pw_name)) { > + strcpy(username, "UNKNOWN"); > + goto fake_it; > + } else { > + pw->pw_passwd = result->sp_pwdp; > + } > +#endif > + > if (pw->pw_passwd[0] == '!' || pw->pw_passwd[0] == '*') > goto auth_failed; > Hi, I wonder if this could be fixed for all applets that use ask_and_check_password (login, su, sulogin, vlock) in libbb/correct_password.c. The following patch is untested (not even compile tested) and only a idea as I haven't touched C code for long time. Ciao, Tito --- libbb/correct_password.c.orig 2020-01-13 00:23:02.432939000 +0100 +++ libbb/correct_password.c 2023-09-13 23:01:40.804878693 +0200 @@ -42,7 +42,7 @@ static const char *get_passwd(const stru { const char *pass; - if (!pw) + if (!pw || pw->pw_passwd[0] == '!') return "aa"; /* "aa" will never match */ pass = pw->pw_passwd; @@ -55,7 +55,7 @@ static const char *get_passwd(const stru * At least glibc 2.4 does this. Be extra paranoid here. */ struct spwd *result = NULL; r = getspnam_r(pw->pw_name, &spw, buffer, SHADOW_BUFSIZE, &result); - pass = (r || !result) ? "aa" : result->sp_pwdp; + pass = (r || !result || result->sp_pwdp[0] == '!' || (result->sp_pwdp[0] == '*' && !result->sp_pwdp[1])) ? "aa" : result->sp_pwdp; } #endif return pass; From Christian.Franke at t-online.de Thu Sep 14 07:53:12 2023 From: Christian.Franke at t-online.de (Christian Franke) Date: Thu, 14 Sep 2023 09:53:12 +0200 Subject: Rename itoa/utoa ? Message-ID: Hi, the existence of the file configs/cygwin_defconfig suggests that there was some upstream support for busybox builds on Cygwin in the ancient past. This does no longer work OOTB since this commit to Cygwin's "newlib" from 2014: https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=32c96dd which added these functions to stdlib.h: ? char * itoa(int value, char * str, int base); ? char * utoa(unsigned value, char * str, int base); These are non-standard and unavailable on Linux and *BSD, but also provided by some other C runtimes (IBM C, MSVC, QNX, ...). MSVC documentation says that these are part of POSIX which is not the case. There is obviously a conflict with the libbb functions: ? char *utoa(unsigned n); ? char *itoa(int n); In my current Cygwin port of busybox-1.36.1 which will appear in Cygwin distro soon I use this ugly but effective hack in libbb.h: +#ifdef __CYGWIN__ +/* Avoid conflict with these non-standard functions from : + * char * itoa(int value, char * str, int base); + * char * utoa(unsigned value, char * str, int base); + */ +# include +# define itoa bb_itoa +# define utoa bb_utoa +#endif Would you accept to rename itoa/utoa to bb_itoa/bb_utoa instead? If there is any interest in the patches for Cygwin, I could eventually rebase these to master and provide a patch series. The current diffstat: $ git diff --stat 1_36_1 ?configs/cygwin_defconfig??????? | 897 +++++++++++++++++++----------- ?coreutils/date.c??????????????? |?? 8 +- ?coreutils/du.c????????????????? |?? 4 + ?coreutils/ls.c????????????????? |?? 3 + ?coreutils/stat.c??????????????? |?? 2 +- ?include/libbb.h???????????????? |? 24 +- ?include/platform.h????????????? |? 17 +- ?libbb/udp_io.c????????????????? |?? 5 + ?networking/hostname.c?????????? |?? 6 + ?networking/ip_icmp_missing.h??? | 162 ++++++ ?networking/ping.c?????????????? |?? 9 +- ?networking/traceroute.c???????? |? 26 +- ?procps/top.c??????????????????? |?? 2 +- ?shell/ash.c???????????????????? |? 17 + ?util-linux/fdisk.c????????????? |? 16 +- Working applets: $ ./busybox --list | wc -l 245 -- Regards, Christian From rachid.koucha at gmail.com Fri Sep 15 10:04:08 2023 From: rachid.koucha at gmail.com (Rachid Koucha) Date: Fri, 15 Sep 2023 12:04:08 +0200 Subject: Shell prompt output Message-ID: Hi, Using busybox 1.34.1, I noticed that ash shell displays the prompt on stdout instead of stderr when CONFIG_FEATURE_EDITING is set. Other shells like bash do not behave this way: the prompt is always on stderr. Moreover, the IEEE standard specifies that the prompt must be written on stderr: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html *PS1* Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion and written to standard error. Regards, Rachid -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christian.Franke at t-online.de Fri Sep 15 15:22:21 2023 From: Christian.Franke at t-online.de (Christian Franke) Date: Fri, 15 Sep 2023 17:22:21 +0200 Subject: [PATCH] build system: don't use local timezone if SOURCE_DATE_EPOCH is specified Message-ID: <2e94bfbc-ccf3-f54a-28da-6cdbdf99ea23@t-online.de> Hi, harmless bug found during reproducible build tests of new busybox-1.36.1-1 packages for Cygwin distro. Workaround: 'export TZ=UTC' during build. -- Regards, Christian -------------- next part -------------- From 93b59db816482f84223feade753b7186fb9540aa Mon Sep 17 00:00:00 2001 From: Christian Franke Date: Fri, 15 Sep 2023 17:16:07 +0200 Subject: [PATCH] build system: don't use local timezone if SOURCE_DATE_EPOCH is specified The ctime() call after gmtime() may overwrite the static 'struct tm' buffer. Signed-off-by: Christian Franke --- scripts/kconfig/confdata.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c index 249a3195e..7ce5b8088 100644 --- a/scripts/kconfig/confdata.c +++ b/scripts/kconfig/confdata.c @@ -382,13 +382,10 @@ int conf_write(const char *name) sym_calc_value(sym); source_date_epoch = getenv("SOURCE_DATE_EPOCH"); - if (source_date_epoch && *source_date_epoch) { + if (source_date_epoch && *source_date_epoch) now = strtoull(source_date_epoch, NULL, 10); - build_time = gmtime(&now); - } else { + else time(&now); - build_time = localtime(&now); - } env = getenv("KCONFIG_NOTIMESTAMP"); if (env && *env) @@ -402,6 +399,13 @@ int conf_write(const char *name) sym_get_string_value(sym), use_timestamp ? "# " : "", use_timestamp ? ctime(&now) : ""); + + /* do this after ctime() because it may overwrite the static 'struct tm' buf */ + if (source_date_epoch && *source_date_epoch) + build_time = gmtime(&now); + else + build_time = localtime(&now); + if (out_h) { char buf[sizeof("#define AUTOCONF_TIMESTAMP " "\"YYYY-MM-DD HH:MM:SS some_timezone\"\n")]; -- 2.39.0 From rob at landley.net Sat Sep 16 16:29:22 2023 From: rob at landley.net (Rob Landley) Date: Sat, 16 Sep 2023 11:29:22 -0500 Subject: [PATCH] testsuite: handle /bin/false being busybox in the start-stop-daemon tests In-Reply-To: <20230808180209.2912321-1-ross.burton@arm.com> References: <20230808180209.2912321-1-ross.burton@arm.com> Message-ID: <2dbcb89c-09eb-082b-b539-52e11d82259d@landley.net> On 8/8/23 13:02, ross.burton at arm.com wrote: > From: Ross Burton > > It's known that the final start-stop-daemon test fails if /bin/false is > actually a busybox symlink. Instead of failing, check if false is > busybox and adapt the expected output to match. > > Signed-off-by: Ross Burton FYI I fixed this in toybox a few years back: https://github.com/landley/toybox/blob/0.8.10/main.c#L249 Rob From rob at landley.net Sun Sep 17 16:09:36 2023 From: rob at landley.net (Rob Landley) Date: Sun, 17 Sep 2023 11:09:36 -0500 Subject: memory increase caused by mounting failure In-Reply-To: References: Message-ID: On 9/5/23 23:27, Leng TouTou wrote: > Hi? > I used the following command to mount repeatedly about 10,000 times > ? $ mount -t squashfs -o loop,thread=0 image /mnt/tmp > This command will not mount successfully, then I found that the memory used by > slab increased by about 10000k. A userspace program can't persistently increase slab memory. That's a kernel issue. If doing it 10,000 times is different from doing it 10 times, that's probably a kernel bug. > So is this normal? In the Busybox source code, I saw that when the mount fails, > del_loop uses LOOP_CLR_FD for cleanup.? > Does LOOP_CLR_FD simply disconnect the loop device from the image, leaving the > loop device still available for reuse?? > Or if I want to completely release a loop device and its occupied memory, do I > need to use LOOP_CTL_REMOVE? How many /dev/loop* devices do you see in your devtmpfs? Either ls /dev/loop* or sudo losetup -l There's only a half-dozen parameters you can set for each loop device: the dev/ino of the file to map to, the offset, length, sector granularity, you can set it read only, and rope the partitioning logic into it. (There used to be a crypto layer you could hook in too, but I think that went away.) Rob From roberto.foglietta at gmail.com Mon Sep 18 07:41:32 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 18 Sep 2023 09:41:32 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? Message-ID: Hi all, I am investigating the Android init procedure (one version, one device, not in general) and I found an interesting line about the initialization of the /dev/urandom (seeding, I suppose). cat /proc/cmdline > /dev/urandom Therefore, I developed a more sophisticated way to do that initialisation: # RAF: seeding the urandom device with some data and a few bits of randomness. # The randomness is put at the beginning of some text data, which is going # to be compressed. It is expected that the whole compressed data will be # way different each time, even if a great part of the input is constant. # Moreover, the size of the randomness changes each time into a range of # [32, 64] bytes, and this adds more unpredictability. Like a hash, the # compression algorithm will produce a way different binary output by just # changing a few bytes and initial conditions. { n=$((33 + ${RANDOM:-15}%32)) dd if=/dev/random bs=$n count=1 2>&1 cat /proc/cmdline /proc/*stat /init* } | pigz -$((1 + n%9))c > /dev/urandom & I wish to ask people here in this m-list, because I know that there are Linux experts here, two questions: 1. initialise the /dev/urandom is necessary? Or the kernel provided itself but is it better? 2. In your opinion, the script above can provide a reasonable unpredictable initialisation? In case the /dev/urandom initialisation is a necessity (or a best practice), does it make sense to add it into busybox as an option or as an application? Best regards, R- From yetanothergeek at gmail.com Mon Sep 18 08:11:08 2023 From: yetanothergeek at gmail.com (Jeff Pohlmeyer) Date: Mon, 18 Sep 2023 03:11:08 -0500 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: On Mon, Sep 18, 2023 at 2:42?AM Roberto A. Foglietta wrote: > In case the /dev/urandom initialisation is a necessity (or a best > practice), does it make sense to add it into busybox as an option or > as an application? If you are able to update to a newer version of busybox, you might want to check out the recently added "seedrng" applet, which seems to be a well-considered means of addressing this issue. You can find a (rather lengthy) discussion here: http://lists.busybox.net/pipermail/busybox/2022-April/089545.html - Jeff From roberto.foglietta at gmail.com Mon Sep 18 09:16:46 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 18 Sep 2023 11:16:46 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: On Mon, 18 Sept 2023 at 10:11, Jeff Pohlmeyer wrote: > > On Mon, Sep 18, 2023 at 2:42?AM Roberto A. Foglietta > wrote: > > > In case the /dev/urandom initialisation is a necessity (or a best > > practice), does it make sense to add it into busybox as an option or > > as an application? > > If you are able to update to a newer version of busybox, you might > want to check out the recently added "seedrng" applet, which seems to > be a well-considered means of addressing this issue. Hi Jeff, thanks for the anwer: redfishos:~ # seedrng seedrng: can't create directory '/var/lib/seedrng': No such file or directory redfishos:~ # mkdir -p /var/lib/seedrng redfishos:~ # seedrng Saving 2048 bits of creditable seed for next boot I think that the app could create the directory path if it does not exist. Moreover an option to write on stdout would be nice to have. > You can find a > (rather lengthy) discussion here: > > http://lists.busybox.net/pipermail/busybox/2022-April/089545.html About this discussion, I have noticed two main points 1. the RNG can't actually be seeded from a shell script, due to the reliance on ioctls and the fact that entropy written into the unprivileged /dev/urandom device is not immediately mixed in, making subsequent seed reads dangerous. 2. I suppose that the kernel will load the generated file in the standard folder at the next boot time without further changes but I am not sure about that. For sure, it will not succeed in my case because rootfs a volatile filesystem and adding a link to a permanent data partition is not a general solution (for this system and at the moment). IMHO, the best I can do is to seed the /dev/urandom by injecting some data and then retrieve some data from it. I have no clue how long the data read from /dev/urandom to be granted that the entropy injected into it will be mixed as expected. I have created a function that generates more than 2048 bytes for seeding the /dev/urandom and read 4Kb after hoping to trigger the mix of the new entropy. udvseed(){ local n=$((33+${RANDOM:-15}%32)) u=/dev/urandom;f(){ dd if=$u bs=$n count=1; };(cd /proc;f;cat cmdline *stat;f;) 2>&1|pigz -$((1+n%9))c >$u; } I wrote it in a way to be short. In fact, it is 153 bytes while the seedrng app in busybox is about 1650. The function and the app are completely different and it is not a fair comparison. However, it is not the first time that I noticed that a busybox app can be easily replaced with a shell script function and this reduces N times the footprint. Best regards, R- From oss-lists at triops.cz Mon Sep 18 09:11:48 2023 From: oss-lists at triops.cz (Ladislav Michl) Date: Mon, 18 Sep 2023 11:11:48 +0200 Subject: [PATCH v2] date: exit with failure when clock_settime fails Message-ID: From: Ladislav Michl Coreutils date behaves this way since 1998-12-11 as done in their git commit a17cdb11731e ("(main): Arrange to exit unsuccessfully when stime fails.") Signed-off-by: Ladislav Michl --- CHANGES: -v2: better compatibily with coreutils, add explaining commit message coreutils/date.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/coreutils/date.c b/coreutils/date.c index 3a89b6caf..09d5697dc 100644 --- a/coreutils/date.c +++ b/coreutils/date.c @@ -166,12 +166,13 @@ int date_main(int argc UNUSED_PARAM, char **argv) struct tm tm_time; char buf_fmt_dt2str[64]; unsigned opt; - int isofmt = -1; char *date_str; char *fmt_dt2str; char *fmt_str2dt; char *filename; char *isofmt_arg = NULL; + int ret = EXIT_SUCCESS; + int isofmt = -1; opt = getopt32long(argv, "^" "Rs:ud:r:" @@ -287,9 +288,12 @@ int date_main(int argc UNUSED_PARAM, char **argv) ts.tv_sec = validate_tm_time(date_str, &tm_time); ts.tv_nsec = 0; - /* if setting time, set it */ + /* if setting time, set the system clock to the specified date, + * then regardless of the success of that operation, + * format and print that date. */ if ((opt & OPT_SET) && clock_settime(CLOCK_REALTIME, &ts) < 0) { bb_simple_perror_msg("can't set date"); + ret = EXIT_FAILURE; } } @@ -383,5 +387,5 @@ int date_main(int argc UNUSED_PARAM, char **argv) } puts(date_buf); - return EXIT_SUCCESS; + return ret; } -- 2.39.2 From guille.rodriguez at gmail.com Mon Sep 18 09:20:25 2023 From: guille.rodriguez at gmail.com (Guillermo Rodriguez Garcia) Date: Mon, 18 Sep 2023 11:20:25 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: El lun, 18 sept 2023 a las 9:42, Roberto A. Foglietta (< roberto.foglietta at gmail.com>) escribi?: > Hi all, > > I am investigating the Android init procedure (one version, one > device, not in general) and I found an interesting line about the > initialization of the /dev/urandom (seeding, I suppose). > > cat /proc/cmdline > /dev/urandom > > Therefore, I developed a more sophisticated way to do that initialisation: > > # RAF: seeding the urandom device with some data and a few bits of > randomness. > # The randomness is put at the beginning of some text data, which is > going > # to be compressed. It is expected that the whole compressed data > will be > # way different each time, even if a great part of the input is > constant. > # Moreover, the size of the randomness changes each time into a range > of > # [32, 64] bytes, and this adds more unpredictability. Like a hash, > the > # compression algorithm will produce a way different binary output by > just > # changing a few bytes and initial conditions. > { > n=$((33 + ${RANDOM:-15}%32)) > dd if=/dev/random bs=$n count=1 2>&1 > cat /proc/cmdline /proc/*stat /init* > } | pigz -$((1 + n%9))c > /dev/urandom & > Not sure whether seeding dev/urandom with output from dev/random makes much sense, since both use the same source of entropy. Also note that dev/random will block if there is not enough entropy left, so doing this in an init script might not be a very good idea -- specially on systems that don't have a good source of entropy available. Guillermo -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.foglietta at gmail.com Mon Sep 18 09:53:41 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 18 Sep 2023 11:53:41 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: On Mon, 18 Sept 2023 at 11:20, Guillermo Rodriguez Garcia wrote: > >> # RAF: seeding the urandom device with some data and a few bits of randomness. >> # The randomness is put at the beginning of some text data, which is going >> # to be compressed. It is expected that the whole compressed data will be >> # way different each time, even if a great part of the input is constant. >> # Moreover, the size of the randomness changes each time into a range of >> # [32, 64] bytes, and this adds more unpredictability. Like a hash, the >> # compression algorithm will produce a way different binary output by just >> # changing a few bytes and initial conditions. >> { >> n=$((33 + ${RANDOM:-15}%32)) >> dd if=/dev/random bs=$n count=1 2>&1 >> cat /proc/cmdline /proc/*stat /init* >> } | pigz -$((1 + n%9))c > /dev/urandom & > Hi Gulliermo, first of all, thank for the feedback. > Not sure whether seeding dev/urandom with output from dev/random makes much sense, since both use the same source of entropy. AFAIK, the /dev/random uses a source of entropy related to hardware events while /dev/urandom is a pseudo-random generator. This should grant us that there is a difference between the two. immediately after a boot, it is supposed that many hardware events took place and therefore reading some bytes from /dev/random would not be such a big issue., IMHO. > Also note that dev/random will block if there is not enough entropy left, so doing this in an init script might not be a very good idea -- specially on systems that don't have a good source of entropy available. As as you might noticed in the function that I sent later answering to Jeff, I do not use anymore the /dev/random but /dev/urandom udvseed(){ local n=$((33+${RANDOM:-15}%32)) u=/dev/urandom;f(){ dd if=$u bs=$n count=1; };(cd /proc;f;cat cmdline *stat;f;) 2>&1|pigz -$((1+n%9))c >$u; } This makes your first statement something to consider. Also $RANDOM if available should be considered generated by the /dev/urandom and therefore belonging to its entropy pool. If $RANDOM is not available then my function is quite weak in term of unpredictability because read 48 bytes from /dev/urandom which is not seeded yet and use it with some proc data that might change but can be guessed and then everything is compressed with gzip -7. This is the worst scenario. However, even in the worst scenario due to gzip one single bit of difference in the input generates a completely different compressed output: redfishos:~ # cat /etc/firmware/touch_module_id_0x82.img | pigz -7c | sha1sum 1f0e7e00a47159708a5877b052d2e2c6e3489788 - redfishos:~ # { cat /etc/firmware/touch_module_id_0x82.img; echo; } | pigz -7c | sha1sum d8db1ae97fc5ac8fa441db5c146d95fc43ac6d2e In the best scenario $RANDOM provides a number that makes the procedure change boot. The data read from /dev/urandom can be 32 bytes or 64 bytes and the compression level can vary between 1 and 9. Therefore every single bit of the input should be correctly guessed otherwise the output will be completely different. Instead of using $RANDOM, it is possible to use a value generated from /dev/random by a single byte read: randval=$(dd if=/dev/random bs=1 count=1 status=none | hexdump -ve '1/1 "%d\n"') Or keep the first variant in which the initial data is read from /dev/random in a range between 32 and 64 bytes, adding this trick just in case $RANDOM is not defined. Best regards, R- From guille.rodriguez at gmail.com Mon Sep 18 10:14:09 2023 From: guille.rodriguez at gmail.com (Guillermo Rodriguez Garcia) Date: Mon, 18 Sep 2023 12:14:09 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: Hi Roberto, El lun, 18 sept 2023 a las 11:54, Roberto A. Foglietta (< roberto.foglietta at gmail.com>) escribi?: > On Mon, 18 Sept 2023 at 11:20, Guillermo Rodriguez Garcia > wrote: > > > >> # RAF: seeding the urandom device with some data and a few bits of > randomness. > >> # The randomness is put at the beginning of some text data, which > is going > >> # to be compressed. It is expected that the whole compressed data > will be > >> # way different each time, even if a great part of the input is > constant. > >> # Moreover, the size of the randomness changes each time into a > range of > >> # [32, 64] bytes, and this adds more unpredictability. Like a > hash, the > >> # compression algorithm will produce a way different binary output > by just > >> # changing a few bytes and initial conditions. > >> { > >> n=$((33 + ${RANDOM:-15}%32)) > >> dd if=/dev/random bs=$n count=1 2>&1 > >> cat /proc/cmdline /proc/*stat /init* > >> } | pigz -$((1 + n%9))c > /dev/urandom & > > > > Hi Gulliermo, > > first of all, thank for the feedback. > > > Not sure whether seeding dev/urandom with output from dev/random makes > much sense, since both use the same source of entropy. > > AFAIK, the /dev/random uses a source of entropy related to hardware > events while /dev/urandom is a pseudo-random generator. No, this is a common myth but it is not correct. Both random and urandom use the same PRNG. See: https://www.2uo.de/myths-about-urandom/ > This should > grant us that there is a difference between the two. immediately after > a boot, it is supposed that many hardware events took place Or not. This could be, for example, a cloned VM running on the cloud (which is a typical use case for busybox). > and > therefore reading some bytes from /dev/random would not be such a big > issue., IMHO. > > > Also note that dev/random will block if there is not enough entropy > left, so doing this in an init script might not be a very good idea -- > specially on systems that don't have a good source of entropy available. > > As as you might noticed in the function that I sent later answering > to Jeff, I do not use anymore the /dev/random but /dev/urandom > > udvseed(){ local n=$((33+${RANDOM:-15}%32)) u=/dev/urandom;f(){ dd > if=$u bs=$n count=1; };(cd /proc;f;cat cmdline *stat;f;) 2>&1|pigz > -$((1+n%9))c >$u; } > > This makes your first statement something to consider. Also $RANDOM if > available should be considered generated by the /dev/urandom and > therefore belonging to its entropy pool. If $RANDOM is not available > then my function is quite weak in term of unpredictability because > read 48 bytes from /dev/urandom which is not seeded yet and use it > with some proc data that might change but can be guessed and then > everything is compressed with gzip -7. This is the worst scenario. > However, even in the worst scenario due to gzip one single bit of > difference in the input generates a completely different compressed > output: > Compression (or any other deterministic manipulation of data) does not add any entropy (or "unpredictability") since the processing is 100% reproducible. In terms of entropy the output of the function is as good (or as bad) as the amount of entropy in the initial seed. Best regards, Guillermo Rodriguez Garcia guille.rodriguez at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mconrad at intellitree.com Tue Sep 19 01:19:38 2023 From: mconrad at intellitree.com (Michael Conrad) Date: Tue, 19 Sep 2023 01:19:38 +0000 (UTC) Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> On 9/18/23 06:14, Guillermo Rodriguez Garcia wrote: > > everything is compressed with gzip -7. This is the worst scenario. > However, even in the worst scenario due to gzip one single bit of > difference in the input generates a completely different compressed > output: > > > Compression (or any other deterministic manipulation of data) does not > add any entropy (or "unpredictability") since the processing is 100% > reproducible. > In terms of entropy the output of the function is as good (or as bad) > as the amount of entropy in the initial seed. Even aside from that, using gzip as some sort of hash function is not going to be anywhere near as good as using an actual hash function, like sha256, sha1 or even md5. I would expect this all goes into the kernel's own hashing inside the RNG and so gzip or any other hash function before delivering it to the kernel is probably irrelevant. The name of the game is to find actually random bits, which you either need to save from the previous boot, or obtain from hardware somehow.? The low bits of thermal sensors and multithreading scheduler timing nanoseconds are probably your best bet if you can't rely on having a hardware entropy generator. -Mike C -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.foglietta at gmail.com Tue Sep 19 05:36:16 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 19 Sep 2023 07:36:16 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> Message-ID: On Tue, 19 Sept 2023 at 03:25, Michael Conrad wrote: > On 9/18/23 06:14, Guillermo Rodriguez Garcia wrote: > > everything is compressed with gzip -7. This is the worst scenario. >> However, even in the worst scenario due to gzip one single bit of >> difference in the input generates a completely different compressed >> output: >> > > Compression (or any other deterministic manipulation of data) does not add > any entropy (or "unpredictability") since the processing is 100% > reproducible. > In terms of entropy the output of the function is as good (or as bad) as > the amount of entropy in the initial seed. > > Hi Michel, > Even aside from that, using gzip as some sort of hash function is not > going to be anywhere near as good as using an actual hash function, like > sha256, sha1 or even md5. > PREMISE Hashing functions and compression algorithms are two completely different kinds of mathematical tools. The most obvious difference is that 1. hash produces an output whose size is fixed whatever is the size of the input while compression output size might vary when input size changes 2. compression algorithms (f) have their counterpart (f??) that reverse the process while hash have not 3. because of point #2 the compression algorithm is bi-univocal functions the same input gave the same output and the same output brings back to the same input 4 We know that hashing functions are always injective functions: the same input gives the same output but the same output can have different input (collisions). Unless a hardware system is provided with a specific hardware component that produces constant entropy (white noise, preferably) the main problem is to create it from few reasonably good random inputs. As you can imagine, we can start a debate about the definition of "reasonably good random inputs" or "entropy". Or at the opposite, we can accept that those definitions are - restricted to our specific sector - simply meaning unpredictable data - unpredictable by an attacker or even better by the root admin of the system. Nanoseconds time granularity cannot be predicted by an attacker and also a system administrator could have a real hard time in doing that without sophisticated external hardware instruments. Unfortunately, not all systems are able to provide a nanosecond timing and the first main reason of this lack depend by the clock frequency: to have nanoseconds granularity (10??) is necessary to have a GHz (10?) clock. MD5SUM, GZIP AND THE WHITE NOISE A relatively weak hash like MD5SUM is way better to create an unpredictable stream of data than any compression function. Ok, let see it: +:git-restore:recovery2:yamui> echo | md5sum 68b329da9893e34099c7d8ad5cb9c940 - As you can see I have 1/16 chance to guess the right next char in the md5sum output. At this point you notice that I am unfair because I used the output of the md5sum command-line (textual human-friendly representation) instead of the md5sum() binary output stream. Obviously, you are right. Hence, I make you notice that you did the same considering the gzip. You took the whole stream which also contains the information to decompress that stream of data. Modifying the function in a way that decompression information are not sent to the output, the output cannot be reversed anymore (f??) does not exist anymore. Because of this trick we can have a sort of length variable hashing function. It is a very bad hash function, because the fixed size of the output is a great feature. A great feature for the primary purpose for which hash functions are currently used, not so great for generating entropy. In fact, if we have 8 bit of entropy, the hash can provide us 512 bit of data stream - white noise - but the number of the 512 bit dataset that could provide us remains 256. In other words: O(8-bit-entropy) = 256 = O(sh512sum 8-bit-entropy). This is what Gulliermo wrote proposing as "entropy conservation principle". The entropy of a closed system never remains constant along the time but always increases and this is a currently accepted principle of physics. Guillermo confused the information with the entropy. However, the principle for which information is an immutable constant is not yet established in physics. I suggest abandoning this kind of consideration and remain confined in our specific sector. In our specific sector, the spectral analysis cannot confute this claim: removed all the data which are specifically tailored for decompressing, the compressed data stream can be statistically separated by white noise in a sensitive way. Well, it is not 100% true. The gzip -1 output can be discriminated by this kind of analysis from gzip -9 output with a certain degree of confidence. However, if we do the same with a real-world entropy generator - say based on thermal effects - we notice a slight rose-noise effect, a tiny-tiny-tiny-tiny predominance in low-frequencies. I used the word tiny 4 times because the black-body law has a 4 as an exponent, but please remember that correlation is not causation... ;-) At this point you may argue that the whole gzip output is made to be reversible and it is totally arbitrary to make a distinction between compressed data and decompressing information. Let me to be more specific, in most of the cases, it is enough to discard the de/compression symbols tables to earn the white-noise badge. The white noise is our rabbit to follow down to Alice's Wonderland hole. CREATING ENTROPY AT OUR WILL Well, I cannot grant that our will is enough but it is a starting point having a good attitude towards the idea that entropy can be multiplied, at least. Hence, let's start with 8 bits of entropy (unpredictable data). A way to increase this entropy in order that O(input) < O(output) cannot be a deterministic function. Gulliermo wrote. Gulliermo is wrong, not totally wrong, tiny-tiny-tiny-tiny wrong. But tiny? is enough for trigger the chaos and leveraging the chaos theory we can multiply the entropy. Therefore, some deterministic functions running on some deterministic systems can multiply entropy. Before the practice, a bit of bare-simple theory, an example rather than a theory. The most sold book in the world is the "Don Quixote" wikipedia states [1] - someone can argue that the most sold book in the world is the Bible - fair, use it if you wish. The main point is - a certain amount of data that is available and knowledgeable to everyone in an easiest way. In double-keys cryptography, it would be the public key. +:git-restore:recovery2:yamui> wget https://www.gutenberg.org/files/996/996-0.txt -qO - | wc -c 2391549 +:git-restore:recovery2:yamui> wget https://www.gutenberg.org/files/996/996-0.txt -qO - | pigz -4c | dd bs=1 of=/dev/null 952365+0 records in +:git-restore:recovery2:yamui> { wget https://www.gutenberg.org/files/996/996-0.txt -qO -; echo Hello World\!; } | pigz -4c | dd bs=1 of=/dev/null 952375+0 records in The ASCII version of the "Don Quixote" is 2391549 while its compression version is 952365. We should account to remove de/compresison tables from this size but for the sake of simplicity, we are going to use these numbers in the example. Implementations might vary. We can also notice that adding a couple of words the compressed data size does not change very much. Now, we can start. 1. we have a byte of entropy 2. we have a public data 2.3M larger than our entropy 3. compressing that data can generates about 1M times bigger data stream than our initial entropy How can we leverage this to multiply our entropy creating a deterministic function running on an almost deterministic system to multiply entropy? Where almost means 100% deterministic - tiny?. The number of characters is 2391549 = 0x247DFD and can be expressed with a 24 bits integer. We can read 3 bytes from the /dev/urandom put in the variable X and X % 2391549 gives us a position. You may argue that this is unfair because 1. /dev/urandom is the system that we need to provide and 2. in its current state is not yet seated properly therefore is predictable. Ok, we have 8 bit of randomness, time to start to use them: randval % 4. Now we have 6 bits + 2 bits. r=6bit, we can read the sort( {r, 63-r, 0x3F xor r} ) occurrences from /dev/urandom and decide their order using the 2 bits. bit0: order or reversed order, bit1: the middle goes in front or remains in the middle. You insist that /dev/urandom cannot be used, ok. sha512sum(randval) = 64 bytes, enough to play the trick and select the 3 bytes. Guillermo is getting mad and screaming that the entropy has remained the same, I just played with bits. He is right, and I am gonna to play even more with the bits. Let me show you. With a 24bit integer, name it N, I can choose a position in the Don Quichotte ASCII text doing N % sizeof(text). In that position, I put my 8 bit of entropy. The compressed data will be different from the original text only after the compression function has met the changed char. Hence we consider the data after that point. That point is a cut-point. This brings a great variance in the size and time of compression. The size is not anymore predictable because I used the entropy to determine where to cut the text and insert the change and because I have inserted my 8 bit of entropy I can also have 256 different outputs. The size and the time still have a kind of correlation with the cutting-point but their least decimals do not. export LAST_NS_TIME=$(nstime); dd if=996-0.txt bs=1 skip=$(($RANDOM$RANDOM$RANDOM$RANDOM % 2391549)) status=none | pigz -4c >output; du -b output; nstime - 293575 output +1.415828601 Now, I have some random noise (239k) and at least a new 8-bit of ture entropy (much probably 16 bits) that I can use for the next round. Because 75 in the size and the 601 in the time are not predictable, at all. You might argue that even removing the de/compression tables a compressed data stream is not a white noise. If it would not, then some characters (frequency) would be much more frequent than others and this means that the compression algorithm is not working very well in compressing data (can be gzip -1 instead of gzip -9) or it is flawed (then we are going to discard it or fix it). I got white noise and I got entropy. TINY?4 VARIATIONS AND CHAOS Up to here we played with unpredictable tiny?? variations to create white noise and a few bits of entropy with a large amount of work and some time. Now it is the time to trigger the chaos and provide us with a large amount of entropy which means elevating the white noise to the entropy status. How? Leveraging tiny?? variation to create totally impredicatable outputs. export LAST_NS_TIME=$(nstime); dd if=996-0.txt bs=1 skip=$(($RANDOM$RANDOM$RANDOM$RANDOM % 2391549)) status=progress 2>&1 | pigz -4c >output; du -b output; nstime - 939672 output +5.748793786 export LAST_NS_TIME=$(nstime); dd if=996-0.txt bs=1 skip=$(($RANDOM$RANDOM$RANDOM$RANDOM % 2391549)) status=progress 2>&1 | pigz -4c >output; du -b output; nstime - 97220 output +1.507189217 With two different runs and 6 seconds, about 1Mb of entropy has been created. Notice that the stderr of the dd progress output is mixed with the stdout that goes into compression input. Do not trust me, do your tests and see with your own eyes that the way in which standard error and standard output mixup is not totally predictable. In general the output of something like this ( thread one >&1 & thread two >&2 & ) 2>&1 is not predictable especially on a SMP system. Imagine that thread one and thread two are just reading the first and the second halves of the Don Chichotte and the third is compressing a random mixing up of these two streams. To spread even better the butter on the bread, adding a usleep in the between using our 8-bit of initial randomness ( thread one >&1 & usleep $randval; thread two >&2 & ) 2>&1 Obviously, the throughput should be slow enough to make the mixture fine grained and bs=1 can give us a reasonable hope about it. IMPLEMENTATION In this function 3 flux of data are going to mix up among them in an unpredictable way: stderr from multiple source, stdout and set -x hwget_random_byte_value() { local dev=/dev/random test -e /dev/.random && dev=/dev/.random dd if=$dev bs=1 count=1 | hexdump -ve '1/1 "%d\n"'; } #endfunc urandom_initseed() { local n m u=/dev/urandom bbcmd=/usr/bin/busybox-static { set -x n=$(hwget_random_byte_value) $bbcmd dd if=$u bs=$((n+32)) count=1; m=$(date +%N); printf "$m" hcmd=$(echo "md5:sha1:sha3:sha256:sha512" | cut -d: -f$((1 + 1${m}%5))) (cd /proc; cat *stat softirqs|tee - >&2|eval ${hcmd}sum;) | $bbcmd dd set +x } 2>&1 | pigz -$((1 + n%9))c | dd bs=$((5+m%64)) skip=1 >$u 2>/dev/null dd if=$u bs=$((1024+n+m%256)) count=1 >/dev/null 2>&1 } #endfunc Removed the de/compression table from the compressed output and we created an entropy generator with a white noise spectral output. On a 2Kb the table is quite small and set at the beginning of the data stream. Therefore skipping something between [5,68] bytes can be enough. However, improvements are always possible and mistakes also. CONCLUSION Our computer systems are not totally deterministic, especially those working at high frequencies of clock (1GHz) and have multiple cores (SMP). These system are almost deterministic with tiny?? variations. Because these systems are digitals, these variations never pop-up on the numbers. Otherwise the ECC memory and CPU checks will trap these errors. Although they provide a deterministic calculation, their functioning is not absolutely deterministic, especially about timings on SMP concurrent processes. This can be leveraged to create large amounts of white noise or triggering the chaos to produce entropy (an unpredictable sequence of numbers) with a white noise spectral profile (the best) in relatively large amounts. NOTE [1] https://en.wikipedia.org/wiki/List_of_best-selling_books -------------- next part -------------- An HTML attachment was scrubbed... URL: From mconrad at intellitree.com Tue Sep 19 06:11:42 2023 From: mconrad at intellitree.com (Michael Conrad) Date: Tue, 19 Sep 2023 06:11:42 +0000 (UTC) Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> Message-ID: <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> On 9/19/23 01:36, Roberto A. Foglietta wrote: > On Tue, 19 Sept 2023 at 03:25, Michael Conrad > wrote: > > On 9/18/23 06:14, Guillermo Rodriguez Garcia wrote: >> >> everything is compressed with gzip -7. This is the worst >> scenario. >> However, even in the worst scenario due to gzip one single bit of >> difference in the input generates a completely different >> compressed >> output: >> >> >> Compression (or any other deterministic manipulation of data) >> does not add any entropy (or "unpredictability") since the >> processing is 100% reproducible. >> In terms of entropy the output of the function is as good (or as >> bad) as the amount of entropy in the initial seed. > > Hi Michel, > > Even aside from that, using gzip as some sort of hash function is > not going to be anywhere near as good as using an actual hash > function, like sha256, sha1 or even md5. > > > PREMISE > > Hashing functions and compression algorithms are two completely > different kinds of mathematical tools. The most obvious difference is that That's a very large response so I'm not sure which part to quote, but I think you're still missing the point.? Assuming your initialization of randomness is meant to stop people from guessing the RSA/SSL keys you generate on a small low-power device during startup scripts, the attacker will probably have a copy of your device and be able to replay any of the steps you take during your startup script.? It will not matter whether you took a few random bits and a large static text and ran it through pigz before feeding it to /dev/urandom, because the attacker will start with guesses of the same few bits and also run them and that same static text through pigz to potentially get the same output.? If they mirror your initialization of /dev/urandom, then they can try to guess your RSA keys.? The only defense is having more actual bits of entropy, or preserving them from a previous boot in a manner that the attacker can't read. My comment about hashing functions was implying that they would be used repeatedly, not that the entire input would be condensed into one single hash.? Yes there is a reduction of entropy if you feed a hash function more than 20 bytes of data and only get 20 out of it.? My point was that no hashing or compression is needed, because the kernel RNG is itself a sort of hash function and should handle that.? Just feed it the raw input bytes. -Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From guille.rodriguez at gmail.com Tue Sep 19 07:10:34 2023 From: guille.rodriguez at gmail.com (Guillermo Rodriguez Garcia) Date: Tue, 19 Sep 2023 09:10:34 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> Message-ID: El martes, 19 de septiembre de 2023, Roberto A. Foglietta < roberto.foglietta at gmail.com> escribi?: > > Guillermo confused the information with the entropy. > > I didn?t confuse anything. Please leave me out of your arguments, thank you. > Guillermo is getting mad and screaming > > :-? You definitely need help. Guillermo -- Guillermo Rodriguez Garcia guille.rodriguez at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam at liddicott.com Tue Sep 19 07:20:29 2023 From: sam at liddicott.com (Sam Liddicott) Date: Tue, 19 Sep 2023 08:20:29 +0100 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: This is also a problem with many virtual appliances. There is no easy answer to this kernel bootup entropy problem and it is lamentable that solutions are delegated to user space, and even to scripts, and to non-experts who will do a bad job of it but not know it. You may as well feed what random data you have into rngd which will supply it to the kernel. You may as well feed /dev/urandom back, as has been pointed out, it is well-hashed (though then stretched) original entropy. It's a very poor solution but if there was a good solution it would be built into the kernel. (I remember the days when random state was saved on shutdown to be re-used on bootup). If you have control over the hardware spec, get a good random device on-board, RPMB or a TPM or something that can produce real random data on demand and provides /dev/hwrng against which you can run rngd to keep entropy full. My particular advice is to not simply dump some limited amount of poor "entropy" into the pool, because in corner cases (especially if retry is needed) it may not always be enough for whatever early processes need it. Instead, use rngd to keep the pool full enough until you get past this stage and then kill rngd. Having poor entropy is bad enough, but stalling boot because you don't have enough poor entropy because you guessed wrong... Some processes unexpectedly consume entropy, e.g. openssl has been seen to consume entropy even when decrypting. If you decrypt enough things you can stall on entropy starvation. Sam On Tue, 19 Sept 2023 at 07:12, Michael Conrad wrote: > On 9/19/23 01:36, Roberto A. Foglietta wrote: > > On Tue, 19 Sept 2023 at 03:25, Michael Conrad > wrote: > >> On 9/18/23 06:14, Guillermo Rodriguez Garcia wrote: >> >> everything is compressed with gzip -7. This is the worst scenario. >>> However, even in the worst scenario due to gzip one single bit of >>> difference in the input generates a completely different compressed >>> output: >>> >> >> Compression (or any other deterministic manipulation of data) does not >> add any entropy (or "unpredictability") since the processing is 100% >> reproducible. >> In terms of entropy the output of the function is as good (or as bad) as >> the amount of entropy in the initial seed. >> >> Hi Michel, > >> Even aside from that, using gzip as some sort of hash function is not >> going to be anywhere near as good as using an actual hash function, like >> sha256, sha1 or even md5. >> > > PREMISE > > Hashing functions and compression algorithms are two completely different > kinds of mathematical tools. The most obvious difference is that > > That's a very large response so I'm not sure which part to quote, but I > think you're still missing the point. Assuming your initialization of > randomness is meant to stop people from guessing the RSA/SSL keys you > generate on a small low-power device during startup scripts, the attacker > will probably have a copy of your device and be able to replay any of the > steps you take during your startup script. It will not matter whether you > took a few random bits and a large static text and ran it through pigz > before feeding it to /dev/urandom, because the attacker will start with > guesses of the same few bits and also run them and that same static text > through pigz to potentially get the same output. If they mirror your > initialization of /dev/urandom, then they can try to guess your RSA keys. > The only defense is having more actual bits of entropy, or preserving them > from a previous boot in a manner that the attacker can't read. > > My comment about hashing functions was implying that they would be used > repeatedly, not that the entire input would be condensed into one single > hash. Yes there is a reduction of entropy if you feed a hash function more > than 20 bytes of data and only get 20 out of it. My point was that no > hashing or compression is needed, because the kernel RNG is itself a sort > of hash function and should handle that. Just feed it the raw input bytes. > > -Mike > _______________________________________________ > busybox mailing list > busybox at busybox.net > http://lists.busybox.net/mailman/listinfo/busybox > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmadeus at codewreck.org Tue Sep 19 08:17:47 2023 From: asmadeus at codewreck.org (Dominique Martinet) Date: Tue, 19 Sep 2023 17:17:47 +0900 Subject: [PATCH] find: fix -xdev -depth (and -delete) Message-ID: <20230919081747.3136748-1-asmadeus@codewreck.org> From: Dominique Martinet find -xdev with -depth would check for same_fs after the subdirectory has been processed (because the check is done in the file/dir action, which is evaluated too late in the -depth case) This renders `find -xdev -delete` useless, as reported in 2012 here: https://bugs.busybox.net/show_bug.cgi?id=5756 The bug report suggested adding an extra hook, which would be required if we were to keep the current xdev approach that allows all filesystems given in argument, but GNU findutils and OpenBSD find actually stop on the first filesystem boundary e.g. for the following tree: $ find test -exec stat --format "%d %n" {} + 27 test 27 test/file 59 test/tmpfs 27 test/tmpfs/bind 27 test/tmpfs/bind/file 59 test/tmpfs/file (Where 'test/tmpfs' is a tmpfs, and 'test/tmpfs/bind' is a bind mount to a neighboring directory in the same filesystem as 'test' -- also tested with a symlink and -follow for openbsd which has no bind mount) Then `find test test/tmpfs -xdev` does not print test/tmpfs/bind/file. This makes the implementation much simpler (although it's a bit ugly to carry the parent st_dev as an argument to the function) and smaller code, and would allow for easy addition of rm/cp --one-file-system if we want to do that later. function old new delta recursive_action1 398 425 +27 parse_params 1660 1670 +10 recursive_action 70 72 +2 fileAction 216 127 -89 find_main 540 442 -98 ------------------------------------------------------------------------------ (add/remove: 0/0 grow/shrink: 3/2 up/down: 39/-187) Total: -148 bytes text data bss dec hex filename 75774 2510 1552 79836 137dc busybox_old 75626 2510 1552 79688 13748 busybox_unstripped --- This has biten me recently and seems to be a known issue for a while. I don't recall seeing any patch for this since I've been on the list so I took a quick stab, please take a look when you have time. file system loops aside, this can be tested on linux with the following "script": ++++++++++++++ cd $(mktemp -d bb-find-xdev.XXXXXX) || exit mkdir tmpfs mount -t tmpfs tmpfs tmpfs mkdir -p a/b/c touch file tmpfs/file a/b/c/file find $PWD -xdev -delete # ^ errors about tmpfs busy and pwd not empty, # but tmpfs/file is left intact and everyting else is gone umount tmpfs find $PWD -xdev -delete +++++++++++++ findutils/find.c | 44 ++-------------------------------------- include/libbb.h | 1 + libbb/recursive_action.c | 13 +++++++++--- 3 files changed, 13 insertions(+), 45 deletions(-) diff --git a/findutils/find.c b/findutils/find.c index 31c9969886f6..a4a6bbc2df91 100644 --- a/findutils/find.c +++ b/findutils/find.c @@ -501,7 +501,6 @@ struct globals { #endif action ***actions; smallint need_print; - smallint xdev_on; smalluint exitstatus; recurse_flags_t recurse_flags; IF_FEATURE_FIND_EXEC_PLUS(unsigned max_argv_len;) @@ -1015,26 +1014,10 @@ static int FAST_FUNC fileAction( struct stat *statbuf) { int r; - int same_fs = 1; - -#if ENABLE_FEATURE_FIND_XDEV - if (S_ISDIR(statbuf->st_mode) && G.xdev_count) { - int i; - for (i = 0; i < G.xdev_count; i++) { - if (G.xdev_dev[i] == statbuf->st_dev) - goto found; - } - //bb_error_msg("'%s': not same fs", fileName); - same_fs = 0; - found: ; - } -#endif #if ENABLE_FEATURE_FIND_MAXDEPTH if (state->depth < G.minmaxdepth[0]) { - if (same_fs) - return TRUE; /* skip this, continue recursing */ - return SKIP; /* stop recursing */ + return TRUE; /* skip this, continue recursing */ } if (state->depth > G.minmaxdepth[1]) return SKIP; /* stop recursing */ @@ -1051,11 +1034,6 @@ static int FAST_FUNC fileAction( return SKIP; } #endif - /* -xdev stops on mountpoints, but AFTER mountpoit itself - * is processed as usual */ - if (!same_fs) { - return SKIP; - } /* Cannot return 0: our caller, recursive_action(), * will perror() and skip dirs (if called on dir) */ @@ -1295,7 +1273,7 @@ static action*** parse_params(char **argv) #if ENABLE_FEATURE_FIND_XDEV else if (parm == OPT_XDEV) { dbg("%d", __LINE__); - G.xdev_on = 1; + G.recurse_flags |= ACTION_XDEV; } #endif #if ENABLE_FEATURE_FIND_MAXDEPTH @@ -1718,24 +1696,6 @@ int find_main(int argc UNUSED_PARAM, char **argv) G.actions = parse_params(&argv[firstopt]); argv[firstopt] = NULL; -#if ENABLE_FEATURE_FIND_XDEV - if (G.xdev_on) { - struct stat stbuf; - - G.xdev_count = firstopt; - G.xdev_dev = xzalloc(G.xdev_count * sizeof(G.xdev_dev[0])); - for (i = 0; argv[i]; i++) { - /* not xstat(): shouldn't bomb out on - * "find not_exist exist -xdev" */ - if (stat(argv[i], &stbuf) == 0) - G.xdev_dev[i] = stbuf.st_dev; - /* else G.xdev_dev[i] stays 0 and - * won't match any real device dev_t - */ - } - } -#endif - for (i = 0; argv[i]; i++) { if (!recursive_action(argv[i], G.recurse_flags,/* flags */ diff --git a/include/libbb.h b/include/libbb.h index eb97a988045d..e8a7b449aef6 100644 --- a/include/libbb.h +++ b/include/libbb.h @@ -514,6 +514,7 @@ enum { ACTION_DEPTHFIRST = (1 << 3), ACTION_QUIET = (1 << 4), ACTION_DANGLING_OK = (1 << 5), + ACTION_XDEV = (1 << 6), }; typedef uint8_t recurse_flags_t; typedef struct recursive_state { diff --git a/libbb/recursive_action.c b/libbb/recursive_action.c index b1c4bfad7ccf..76dd664369a2 100644 --- a/libbb/recursive_action.c +++ b/libbb/recursive_action.c @@ -62,9 +62,11 @@ static int FAST_FUNC true_action(struct recursive_state *state UNUSED_PARAM, * ACTION_FOLLOWLINKS mainly controls handling of links to dirs. * 0: lstat(statbuf). Calls fileAction on link name even if points to dir. * 1: stat(statbuf). Calls dirAction and optionally recurse on link to dir. + * + * If ACTION_XDEV, stop on different filesystem _after_ it has been processed */ -static int recursive_action1(recursive_state_t *state, const char *fileName) +static int recursive_action1(recursive_state_t *state, const char *fileName, dev_t parentDev) { struct stat statbuf; unsigned follow; @@ -114,6 +116,10 @@ static int recursive_action1(recursive_state_t *state, const char *fileName) return TRUE; } + /* skip cross devices -- we still need to process action */ + if ((state->flags & ACTION_XDEV) && parentDev != 0 && statbuf.st_dev != parentDev) + goto skip_recurse; + dir = opendir(fileName); if (!dir) { /* findutils-4.1.20 reports this */ @@ -132,7 +138,7 @@ static int recursive_action1(recursive_state_t *state, const char *fileName) /* process every file (NB: ACTION_RECURSE is set in flags) */ state->depth++; - s = recursive_action1(state, nextFile); + s = recursive_action1(state, nextFile, statbuf.st_dev); if (s == FALSE) status = FALSE; free(nextFile); @@ -146,6 +152,7 @@ static int recursive_action1(recursive_state_t *state, const char *fileName) } closedir(dir); +skip_recurse: if (state->flags & ACTION_DEPTHFIRST) { if (!state->dirAction(state, fileName, &statbuf)) goto done_nak_warn; @@ -177,5 +184,5 @@ int FAST_FUNC recursive_action(const char *fileName, state.fileAction = fileAction ? fileAction : true_action; state.dirAction = dirAction ? dirAction : true_action; - return recursive_action1(&state, fileName); + return recursive_action1(&state, fileName, 0); } -- 2.39.2 From asmadeus at codewreck.org Tue Sep 19 08:11:02 2023 From: asmadeus at codewreck.org (Dominique Martinet) Date: Tue, 19 Sep 2023 17:11:02 +0900 Subject: [PATCH resend] sed: check errors writing file with sed -i Message-ID: <20230919081102.3136547-1-asmadeus@codewreck.org> From: Dominique Martinet sed would currently not error if write failed when modifying a file. This can be reproduced with the following 'script': $ sudo mount -t tmpfs tmpfs -o size=1M /tmp/m $ sudo chmod 777 /tmp/m $ echo foo > /tmp/m/foo $ dd if=/dev/zero of=/tmp/m/fill bs=4k dd: error writing '/tmp/m/fill': No space left on device 256+0 records in 255+0 records out 1044480 bytes (1.0 MB, 1020 KiB) copied, 0.00234567 s, 445 MB/s $ busybox sed -i -e 's/.*/bar/' /tmp/m/foo $ echo $? 0 $ cat /tmp/m/foo new behaviour: $ echo foo > /tmp/m/foo $ ./busybox sed -i -e 's/.*/bar/' /tmp/m/foo sed: write error $ echo $? 4 $ cat /tmp/m/foo foo function old new delta sed_main 754 801 +47 ------------------------------------------------------------------------------ (add/remove: 0/0 grow/shrink: 1/0 up/down: 47/0) Total: 47 bytes text data bss dec hex filename 75727 2510 1552 79789 137ad busybox_old 75774 2510 1552 79836 137dc busybox_unstripped Signed-off-by: Dominique Martinet --- Resending this patch again as it doesn't seem to have been applied. FWIW it's still applied on alpine builds (since Nov last year); would be great to see this merged so I can forget about it :) editors/sed.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/editors/sed.c b/editors/sed.c index 00dde60be329..6179c5e80958 100644 --- a/editors/sed.c +++ b/editors/sed.c @@ -1648,6 +1648,11 @@ int sed_main(int argc UNUSED_PARAM, char **argv) fchown(nonstdoutfd, statbuf.st_uid, statbuf.st_gid); process_files(); + fflush(G.nonstdout); + if (ferror(G.nonstdout)) { + xfunc_error_retval = 4; /* It's what gnu sed exits with... */ + bb_simple_error_msg_and_die(bb_msg_write_error); + } fclose(G.nonstdout); G.nonstdout = stdout; -- 2.39.2 From rob at landley.net Tue Sep 19 11:33:18 2023 From: rob at landley.net (Rob Landley) Date: Tue, 19 Sep 2023 06:33:18 -0500 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: On 9/19/23 02:20, Sam Liddicott wrote: > This is also a problem with many?virtual appliances. There is no easy answer to > this kernel bootup entropy problem and it is lamentable that solutions are > delegated to user space, and even to scripts, and to non-experts who will do a > bad job of it but not know it. > > You may as well feed what random data you have into rngd which will supply it to > the?kernel. You may as well feed /dev/urandom back, as has been pointed out, it > is well-hashed (though then stretched) original entropy. If it's not initialized by anything you get the same results each run, which generates the same keys during boot, which makes cryptographic attacks on recorded sessions way easier. The only source of variance is the battery backed up clock (when there is one, not all hardware...) and timing jitter in a boot cycle you're trying to make as fast as possible (the record is something like 2/3 of a second). Other historical sources like the network card's mac address stopped being mixed in by drivers because "I am nomad, I am perfect, you must be eliminated" reasoning. (So userspace had to do it via "ifconfig -a > /dev/random" and friends... which is exactly what they nerfed, because nomad.) Imagine a mass produced 25 cent board with basically a rom image and no battery (batteries die, batteries leak acid, batteries age poorly when heated, batteries cost money and take up space), which netboots via public key cryptography (or even just fetches provisioning data that won't fit in 1k of cheap on-chip flash). If it can't generate session keys right after boot, there's a problem. Modern "software" entropy collection is mostly timing jitter collected very slowly as the system runs. The theory for embedded devices was if you have some writeable space in the device you save a few hundred bytes of /dev/urandom output on shutdown and write it back on bootup so you're keeping enough entropy across reboots to seriously salt your attack surface. The perfect is the enemy of the good: this just makes different boots behave differently. I honestly don't understand what creating a new API for this is supposed to do. Supplying the information via ioctl is superior to supplying the same information via /dev or /sys how exactly? Rob From roberto.foglietta at gmail.com Tue Sep 19 12:06:18 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 19 Sep 2023 14:06:18 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: On Tue, 19 Sept 2023 at 13:30, Rob Landley wrote: > > Modern "software" entropy collection is mostly timing jitter collected very > slowly as the system runs. The theory for embedded devices was if you have some > writeable space in the device you save a few hundred bytes of /dev/urandom > output on shutdown and write it back on bootup so you're keeping enough entropy > across reboots to seriously salt your attack surface. The perfect is the enemy > of the good: this just makes different boots behave differently. Even without a small writable space the problem could be addressed because the time jittering at boot kernel time before the system would be ready and expose any kind of service. A general solution that can be completely safe and efficient for all kinds of systems or hardware on which Linux kernel can run is very hard to find. For a subset of systems - fortunately the most used and common to be found on the market - the solution is to amplify the small amount of entropy generated by kernel boot timing jittering and use it to seed the /dev/urandom. Because the amplification process can always be based on mixing multi-task outputs in a single stream in a way that tiny?? timings variations (jittering) makes a difference also in lines ordering - in an truly unpredictable way - then the dmesg output provided with a fine-grained time resolution timestamp on each line can be used as input for a hash function or a compression algorithm tailored for discarding de/compression metadata. With this data it is possible to seed the /dev/random. Even In a system that always start with UNIX EPOCH = 0 and is missing an RTC, in the /init execute this instruction dmesg | sha512sum as soon as possible. I bet that you will never find two occurrences that match as far as the dmesg is provided with a enough fine-grained time resolution. In those system for which the time resolution provided by dmseg is too large compared with the timings jittering at boot time, then the sha512sum will ALWAYS provide the same output unless multi-tasking boot mix-up some lines because hardware initialisation timings can varying enough (a mixed case, few or some hashes are possible but not a great variety of them). In all these cases, seeding the /dev/urandom with this procedure DOSN'T hurt. Is the seed constant? At least it is not the same for all the systems but characteristic of that specific hw/vt instance. Is the seed weak because it can spawn among few/some others, at least is not totally constant like providing nothing at all. The seed is good but the cat /proc/cmdline happens like in Android just because they need to do something about? Fine, we cannot prevent it. As you can see the problem is not the problem, the problem is the perception of the problem. dmesg | sha512sum > /dev/urandom As the first init instruction or as the last kernel boot operation, is THE general answer also when it is not the solution. Some systems need a more dedicated effort but in no way THE answer puts them in a worse condition than they were before. Confutations are welcome. :-) From roberto.foglietta at gmail.com Tue Sep 19 12:20:13 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 19 Sep 2023 14:20:13 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: On Tue, 19 Sept 2023 at 14:06, Roberto A. Foglietta wrote: > > On Tue, 19 Sept 2023 at 13:30, Rob Landley wrote: > dmesg | sha512sum > /dev/urandom Oops, sorry because the sha512sum command-line prints a human readable 16 chars based string dmesg | pigz -4c | dd bs=64 skip=1 > /dev/urandom is a far better option. Obviously, in the kernel sha512sum() provides a binary stream not a text. The good habits to have a basic /dev populated with those few devices that always exist like /dev/urandom allows the seeding even before the mount procedure. Best regards, R- From roberto.foglietta at gmail.com Tue Sep 19 13:34:00 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Tue, 19 Sep 2023 15:34:00 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: On Tue, 19 Sept 2023 at 14:06, Roberto A. Foglietta wrote: > > On Tue, 19 Sept 2023 at 13:30, Rob Landley wrote: > As the first init instruction or as the last kernel boot operation, is > THE general answer also when it is not the solution. Some systems need > a more dedicated effort but in no way THE answer puts them in a worse > condition than they were before. Confutations are welcome. :-) Yes, it hurts because on some specific systems the /dev/urandom is filled with a data stream which is constant for each boot or not good enough for seeding it in a secure way. This means that any later effort wil not have an immediate effect because of that initial seeding. Fine, then use > (write) to reset the entropy and >> (append) to add entropy. This will break the back-compatibility with the past and mess-up all the scripts or tools which are in the production nowadays. Fine, then use > (write) to add entropy as usual and >> (append) to reset entropy. Unfortunately this approach leaves behind some specific systems, when they will update the kernel. At that future time, they will choose to deselect the default option to seed the /dev/random OR use >> to overcome the issue. Hoping that those systems have a better way to seed the /dev/urandom engine which can not be the case at all because a low-frequency 1-single core NOMMU system, possibly with "none" as I/O and task scheduler can have a very small quantity of entropy and no means to amplify it. We may agree that those systems - because of their intrinsic limitations - are not sophisticated enough for being exposed in an untrustworthy environment like the Internet or they should integrate/connect a I2C/USB cheap thermal sensor that can provide true white noise, for example. With 1% of the effort, 99% of the work is done. The rest should be dealt with by experts which will select the more appropriate set of features and options case by case. Trying to deal with all the cases including that 1% lead to an impasse. This is wrong because a general solution should not necessarily be a mathematical truth but a the main way for almost all the cases which includes a reasonable exception handling that allows to deal properly also with the un-usual cases. This is a general solution. A mathematical truth is often useless or counterproductive in the real-world. Math says it is wrong in the most general case while practice says it works in almost all cases. Then add an exception handling and go with the practice. ;-) I hope this helps, R- From David.Laight at ACULAB.COM Tue Sep 19 13:37:07 2023 From: David.Laight at ACULAB.COM (David Laight) Date: Tue, 19 Sep 2023 13:37:07 +0000 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> Message-ID: <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> > Fine, then use > (write) to reset the entropy and >> (append) to add entropy. IIRC writing to /dev/urandom doesn't do what you want it to do. You have to use an ioctl() to actually set entropy. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) From ska-dietlibc at skarnet.org Tue Sep 19 14:14:14 2023 From: ska-dietlibc at skarnet.org (Laurent Bercot) Date: Tue, 19 Sep 2023 14:14:14 +0000 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: Oh boy. Is it that time of the year again already? >In case the /dev/urandom initialisation is a necessity (or a best >practice), does it make sense to add it into busybox as an option or >as an application? So, there are two different things to address here: 1. the way to add entropy to the kernel's entropy pool in order to make /dev/urandom sufficiently unpredictable; 2. the best way to do this as part of a boot process. For the first thing, the problem is that writing to /dev/urandom adds data to the seed, but does not update the entropy count, because the data you write is not considered creditable (i.e. does not come from a source that is considered random enough to warrant increasing the entropy count). That is the reason for the ioctl: to mark some random data as creditable. That is (a part of) why Jason, who revamped the entire random subsystem in the Linux kernel, wrote seedrng[1]. (Unfortunately, he did not approve the modifications to the version of seedrng that made it into busybox, so use bb seedrng at your own risk. I have an alternative implementation of the same concept, rngseed[2], that follows Jason's design more closely, and has additional options to give more flexibility for e.g. systems with read-only boot disks.) The answer's to Roberto's first question is: yes, initializing /dev/urandom is necessary, but writing stuff to /dev/urandom will not help, even if you consider that stuff random enough. You need a specific tool like seedrng. For the second thing: most of the initialization of a system can happen while the seeding of the entropy pool is in progress. However, at some point, you need a good source of randomness, e.g. when starting an sshd server, and you should have a tool that makes sure the entropy pool is full *before* important services start using it to get their random data. seedrng, or rngseed, fill that role. Writing data to /dev/urandom does not. So the answer to Roberto's second question is: no, the provided script excerpt is *not* suitable for seeding the entropy pool, no matter how much compression, or even how much hashing, you use. On that point, busybox cannot help here. busybox has an implementation of seedrng, but it does not provide a full boot sequence or policy whatsoever; it cannot tell *when* during boot it is appropriate to call seedrng. It is all up to system integrators to write boot sequences that seed the entropy pool correctly and start services securely. [1]: https://git.zx2c4.com/seedrng/about/ [2]: https://skarnet.org/software/s6-linux-utils/rngseed.html -- Laurent From steffen at sdaoden.eu Tue Sep 19 18:54:53 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Tue, 19 Sep 2023 20:54:53 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> Message-ID: <20230919185453.WtzGc%steffen@sdaoden.eu> David Laight wrote in <1628d46df2fb4580b474fd0ea43a5778 at AcuMS.aculab.com>: |> Fine, then use > (write) to reset the entropy and >> (append) to \ |> add entropy. | |IIRC writing to /dev/urandom doesn't do what you want it to do. |You have to use an ioctl() to actually set entropy. And that is the sad point about it. Kernel hackers should stand up to allow it again! It wins nothing to only allow this, sorry, grazy lock/ioctl dance unless you have a super-parallel boot sequence, but maybe even then not (as the device below cares, surely it does, not looking). On my (CRUX) distro with SysV style init everything is synchronous during boot/shutdown, and i fail to see the improvement. The kernel should simply agree to have entropy. (This rhymes!!) --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From steffen at sdaoden.eu Tue Sep 19 19:38:05 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Tue, 19 Sep 2023 21:38:05 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: Message-ID: <20230919193805.QftQM%steffen@sdaoden.eu> Laurent Bercot wrote in : ... | The answer's to Roberto's first question is: yes, initializing |/dev/urandom is necessary, but writing stuff to /dev/urandom will not |help, even if you consider that stuff random enough. You need a specific |tool like seedrng. No. ... | For the second thing: most of the initialization of a system can happen |while the seeding of the entropy pool is in progress. However, at some |point, you need a good source of randomness, e.g. when starting an sshd |server, and you should have a tool that makes sure the entropy pool is |full *before* important services start using it to get their random |data. | | seedrng, or rngseed, fill that role. Writing data to /dev/urandom does |not. So the answer to Roberto's second question is: no, the provided Why not? _Only_ by definition. The definition is not right. |script excerpt is *not* suitable for seeding the entropy pool, no matter |how much compression, or even how much hashing, you use. That .. i agree with. (I have not really looked i must admit. This is both truly hairy and totally "exaggerated", in my opinion, sorry for the bad english. I have read OpenBSD's as well as Donenfeld's first as well as Tso's random stuff in the past. If anyone wants to know, in my opinion counting entropy was and is a miracle to me. NetBSD's CVS HEAD now has a truly sophisticated approach that is user tunable, in sofar: nice! I myself say: it is best effort, treat it as "seeded" if stat("/dev/random")->st_blksize (aka "stat -c %o /dev/random") bytes have been written to "it", uh, that is a large value!, and mix in "jitterentropy" and interrupts and what not for sources in the kernel to be unique.) That VM-fork stuff of Jason Donenfeld is a good thing! ... --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From ska-dietlibc at skarnet.org Tue Sep 19 21:58:05 2023 From: ska-dietlibc at skarnet.org (Laurent Bercot) Date: Tue, 19 Sep 2023 21:58:05 +0000 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <20230919185453.WtzGc%steffen@sdaoden.eu> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: >|IIRC writing to /dev/urandom doesn't do what you want it to do. >|You have to use an ioctl() to actually set entropy. > >And that is the sad point about it. >Kernel hackers should stand up to allow it again! As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that any user can write to /dev/urandom, including malicious users, so you cannot credit what they write. I tend to trust people who do the work rather than those who stand on cardboard boxes. Still, since it's related to boot sequence things and I want to be a reliable source on boot sequences, I actually studied the thing when it came up, and understood the issue enough to come up with my own conclusion - and my own conclusion is still that the person who did the work, i.e. Jason, is right about this. I'm sorry. I like the idea of writing stuff to /dev/urandom and have it count, too. It's just not a good idea for security. That's just the way it is. And it would be nice if all the work and ink that already went into it, including mine, could actually be useful to all the people who don't care about any of this and just want their systems to work and be secure - so it would be nice if disinformation and bad ideas stopped being spread. [1]: https://lwn.net/ml/linux-kernel/YjqVemCkZCU1pOzj at mit.edu/ [2]: https://lwn.net/ml/linux-kernel/YjqbcQbYHCOpgqGg at zx2c4.com/ -- Laurent From steffen at sdaoden.eu Tue Sep 19 22:29:10 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Wed, 20 Sep 2023 00:29:10 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: <20230919222910.pn44Y%steffen@sdaoden.eu> Laurent Bercot wrote in : | |>|IIRC writing to /dev/urandom doesn't do what you want it to do. |>|You have to use an ioctl() to actually set entropy. |> |>And that is the sad point about it. |>Kernel hackers should stand up to allow it again! | | As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that |any user can write to /dev/urandom, including malicious users, so |you cannot credit what they write. Well i was talking on unlocking the thing upon boot. Only root there is. And then you could easily look who is the writer, or require CAP_SYS_ADMIN or whatever when deciding whether "entropy" is to be counted or not. That is just a check that is done a thousand times in the kernel, i would think. Actually, if i recall correctly, it is only for unlocking after the rewrite anyway since entropy is not counted no more? At least the IOCTL which gives the number does not change no more. The thing is plain. Unix "everything is a file" ("except when it isn't"; except on Plan9, where it is) is broken by that for no reason. I did not read the files it is too late, but i would have stood up and have done it like that. | I tend to trust people who do the work rather than those who stand on |cardboard boxes. | Still, since it's related to boot sequence things and I want to be a |reliable source on boot sequences, I actually studied the thing when |it came up, and understood the issue enough to come up with my own |conclusion - and my own conclusion is still that the person who did the |work, i.e. Jason, is right about this. | | I'm sorry. I like the idea of writing stuff to /dev/urandom and have |it count, too. It's just not a good idea for security. That's just the |way it is. And it would be nice if all the work and ink that already |went into it, including mine, could actually be useful to all the people |who don't care about any of this and just want their systems to work |and be secure - so it would be nice if disinformation and bad ideas |stopped being spread. What a sheer nonsense. Sorry. |[1]: https://lwn.net/ml/linux-kernel/YjqVemCkZCU1pOzj at mit.edu/ |[2]: https://lwn.net/ml/linux-kernel/YjqbcQbYHCOpgqGg at zx2c4.com/ --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From roberto.foglietta at gmail.com Wed Sep 20 05:06:42 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 20 Sep 2023 07:06:42 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: On Tue, 19 Sept 2023 at 16:20, Laurent Bercot wrote: > > > Oh boy. Is it that time of the year again already? Vote for me, it will be Christmas all the days of the year! *ROTFL* On Tue, 19 Sept 2023 at 23:58, Laurent Bercot wrote: > > >|IIRC writing to /dev/urandom doesn't do what you want it to do. > >|You have to use an ioctl() to actually set entropy. > > > >And that is the sad point about it. > >Kernel hackers should stand up to allow it again! > > As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that > any user can write to /dev/urandom, including malicious users, so > you cannot credit what they write. > Not every users, those have the permission to write on /dev/urandom ~/j/hr/ramdisk# ls -al /dev/urandom crw-rw-rw- 1 root root 1, 9 Sep 17 23:46 /dev/urandom Usually the allowed user is only root which can execute busybox seedrng as well and it can successfully deliver an ioctl() because it has the privileges to do that otherwise not. The sensitive difference is that seedrng is supposed to deliver good enough random data to the entropy pool and who escalate its running privilege should also tamper the data. However, who can escalate the seeding privilege and be able to tamper its data, can also more easily open a root shell and at that point everything can happen. Therefore the idea for which we need a ioctl() to credit data into the entropy pool - AFAIK, it just prevents a mismatch or a poorly /dev/urandom permission settings - IMHO, instead of a ioctl() I would check the permissions and accept as creditable entropy only when the device /dev/urandom is properly configured (or alternatively in a /sys instance if the kernel driver cannot control the /dev/urandom permission settings). Another point in favour of a iotcl() is a write made for mistake by root. S/he wants write test data on /tmp/urandom but s/he type /dev/urandom. Let me say, good luck to all those who wish to ride on this path because preventing root mistakes is an endless and counterproductive way to waste our life-time especially at the kernel level. On Wed, 20 Sept 2023 at 00:29, Steffen Nurpmeso wrote: > > The thing is plain. Unix "everything is a file" ("except when it > isn't"; except on Plan9, where it is) is broken by that for no > reason. I did not read the files it is too late, but i would have > stood up and have done it like that. > Instead, this is the best reason because ioctl()s are bad, in general and in particular in this specific case. === However the main question was: why does the kernel not properly initialize the /dev/urandom entropy pool at boot time considering that 99% of the system generates enough entropy at boot time? The scripts were a way to show in practice (PoC) how we can generate that entropy. By the way, seedrng generate a files which is supposed to be written in a persistent filesystem and used by the kernel at the time of the next boot. I wish you notice two facts about this way of doing things: 1. that credit data can be tampered by an attacker that can escalate the priviledges to write on /dev/urandom and 2. the idea that the kernel read a file system file in a certain position is far more complex than hash some chukns of the boot log containing a fine-timing granularity timestamps into the pool of the /dev/urandom entropy. Finally, repeat one more time all together: ioctls() are bad because they are a trick to workaround the limitation of "in UNIX everything is a file" principle and like every principle it establish some limitations for a good reason. Again, again... Best regards, R- From roberto.foglietta at gmail.com Wed Sep 20 05:51:29 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 20 Sep 2023 07:51:29 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: On Wed, 20 Sept 2023 at 07:06, Roberto A. Foglietta wrote: > Finally, repeat one more time all together: ioctls() are bad because > they are a trick to workaround the limitation of "in UNIX everything > is a file" principle and like every principle it establish some > limitations for a good reason. Again, again... The reason because it exists the ioctl() is well explained here https://stackoverflow.com/questions/17118705/using-rndaddentropy-to-add-entropy-to-dev-random which refers to the man page https://man7.org/linux/man-pages/man4/random.4.html Therefore we can assume that interface is stable and widely accepted while the comments in stackoverflow add an hint: <> - This means that adding good white noise to /dev/urandom would be sufficient to avoid it running without a source of entropy. In my own embedded system /dev/random is moved to /dev/.random and /dev/urandom takes its place. It is a highly-questionable choice, I know. This prevents anything blocking the functioning in the hope that /dev/urandom will be good enough to replace /dev/random. To sustain this conjecture, every 15 minutes, I feed the /dev/urandom with data that I consider good quality white noise. If /dev/random exists, it is supposed to exist for a good reason and replacing it is a questionable approach. I accept it. The way in which /dev/random is used in practice is another story and in my embedded system the SSH connection is a full-day working service while almost all peripherals are inactive or disabled. Much more than a server with disks and not just everything in RAM, it risks running low on entropy very fast. However, SSH is configured to use /dev/urandom by default therefore it is not a sink of entropy and probably there is no any sink of entropy at all. Back to the ioctl() which probably has been introduced before the /sys exist, now the /sys exist. Time to think about planning to replace that ioctl() with a file in /sys? #define BUFSIZE 256 /* WARNING - this struct must match random.h's struct rand_pool_info */ typedef struct { int bit_count; /* number of bits of entropy in data */ int byte_count; /* number of bytes of data in array */ unsigned char buf[BUFSIZ]; } entropy_t; Giving a look to the structure does not seem straightforward to replace it with a file-interface in particular about bit_count. However, that value is ignored for /dev/urandom. Moreover, everything in the buffer will be added after being passed through a hash function. I hate this interface because the main idea behind this interface is clearly: "you are stupid and therefore I prevent you to do stupid things". This is the opposite of the UNIX paradigm. You write a file? I assume that you know what you are doing otherwise expect to meet your Karma soon. Best regards, R- From yetanothergeek at gmail.com Wed Sep 20 15:16:13 2023 From: yetanothergeek at gmail.com (Jeff Pohlmeyer) Date: Wed, 20 Sep 2023 10:16:13 -0500 Subject: syslogd problem Message-ID: It seems that local logging with syslogd is no longer working for me. I think commit 02378ce20c6d2df062357b6d60fc440609d203be (syslogd: decrease stack usage, ~50 bytes) has broken something. After some digging, I noticed this: On lines #1048-1049 : if (ENABLE_FEATURE_REMOTE_LOG && !(opts & OPT_remotelog)) // -R option_mask32 |= OPT_locallog; But on line #1181 : if (!ENABLE_FEATURE_REMOTE_LOG || (opts & OPT_locallog)) { It looks like the code is setting OPT_locallog in "option_mask32" but checking for it in "opts" If I change line #1181 like this: - if (!ENABLE_FEATURE_REMOTE_LOG || (opts & OPT_locallog)) { + if (!ENABLE_FEATURE_REMOTE_LOG || (option_mask32 & OPT_locallog)) { Then it works as expected. - Jeff From kryn at in2p3.fr Wed Sep 20 16:30:39 2023 From: kryn at in2p3.fr (Didier Kryn) Date: Wed, 20 Sep 2023 18:30:39 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: Le 20/09/2023 ? 07:51, Roberto A. Foglietta a ?crit?: > The reason because it exists the ioctl() is well explained here > > https://stackoverflow.com/questions/17118705/using-rndaddentropy-to-add-entropy-to-dev-random > > which refers to the man page > > https://man7.org/linux/man-pages/man4/random.4.html ??? Beware: I may have misunderstood something but the last reference must be outdated since it recommends to save the seed under /var/run ... In Debian/Devuan, /var/run points to /run which is mounted on tmpfs and therefore vanishes on shutdown. Better store the seed under /var/lib. ??? It is the rationale of everything /run to be lost on reboot. ??? /var/lib is not so well defined; it contains stuff which must be preserved (eg hardware clock frequency adjustment) and things which don't need (eg file locks). -- ??? Didier From mjt at tls.msk.ru Wed Sep 20 16:54:56 2023 From: mjt at tls.msk.ru (Michael Tokarev) Date: Wed, 20 Sep 2023 19:54:56 +0300 Subject: syslogd problem In-Reply-To: References: Message-ID: <2b869ce9-af91-81ab-038d-ba89ee7b334e@tls.msk.ru> 20.09.2023 18:16, Jeff Pohlmeyer wrote: > It seems that local logging with syslogd is no longer working for me. > > I think commit 02378ce20c6d2df062357b6d60fc440609d203be > (syslogd: decrease stack usage, ~50 bytes) > has broken something. It's been broken for quite some time http://lists.busybox.net/pipermail/busybox/2023-June/090363.html /mjt From roberto.foglietta at gmail.com Wed Sep 20 18:13:12 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Wed, 20 Sep 2023 20:13:12 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> Message-ID: On Wed, 20 Sept 2023 at 18:52, Didier Kryn wrote: > > It is the rationale of everything /run to be lost on reboot. > Thanks Didier, for having highlighted this. Currently, everything is on RAM in the embedded system I am working on. From steffen at sdaoden.eu Thu Sep 21 00:34:26 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Thu, 21 Sep 2023 02:34:26 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <20230919222910.pn44Y%steffen@sdaoden.eu> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> Message-ID: <20230921003426.-jVMT%steffen@sdaoden.eu> Steffen Nurpmeso wrote in <20230919222910.pn44Y%steffen at sdaoden.eu>: |Laurent Bercot wrote in | : ||>|IIRC writing to /dev/urandom doesn't do what you want it to do. ||>|You have to use an ioctl() to actually set entropy. ||> ||>And that is the sad point about it. ||>Kernel hackers should stand up to allow it again! || || As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that ||any user can write to /dev/urandom, including malicious users, so ||you cannot credit what they write. | |Well i was talking on unlocking the thing upon boot. Only root |there is. | |And then you could easily look who is the writer, or require |CAP_SYS_ADMIN or whatever when deciding whether "entropy" is to be |counted or not. That is just a check that is done a thousand |times in the kernel, i would think. | |Actually, if i recall correctly, it is only for unlocking after |the rewrite anyway since entropy is not counted no more? At least |the IOCTL which gives the number does not change no more. | |The thing is plain. Unix "everything is a file" ("except when it |isn't"; except on Plan9, where it is) is broken by that for no |reason. I did not read the files it is too late, but i would have |stood up and have done it like that. ... ||[1]: https://lwn.net/ml/linux-kernel/YjqVemCkZCU1pOzj at mit.edu/ ||[2]: https://lwn.net/ml/linux-kernel/YjqbcQbYHCOpgqGg at zx2c4.com/ Now i have read them and [2] indeed brings SYS_ADMIN into play but disregards it with The problem with that approach, though, is that various userspaces might already write garbage into /dev/urandom, not expecting them to be credited -- for example, some userspace hardware configuration component that writes some serial number there. So I'm sort of hesitant to _change_ the behavior that we've had for so long. and Another variation on that would be to do what this current patch does, but only crng_pre_init_inject() on CAP_SYS_ADMIN. But this has the same pitfall of only working as intended at cnrg_init=0 but not crng_init=1. One could have made a survey? Before breaking all non-systemd distributions i mean. Change creditable root writes to /dev/random instead? But regardless, they silently broke random in 2019 (due to RANDOM_TRUST_CPU addition) so that my VM hang, so i wrote entropy-saver.c to actually count the entropy i saved/restored myself. Before i had to install haveged because python2 via mailman consumes so much random you get grazy. At least that nonsense concept is now history. Or, as Mr. Donenfeld wrote to me in private after the thread on this list last year, "If you're trying to find some kind of "cryptographically secure cut-off point" instead, what you want is 256 bits / 32 bytes" (instead of 512 bytes). So to be out, i think that is terrible and i like how OpenBSD does it, via shell, aka there the CRUX Linux /etc/rc(.shutdown)? way of doing things would simply continue to work. I think the Blake2/xy approach in the kernel and the VM fork detection and reseeding is great. Getting rid of entropy count is great. (All of that seems to have been massively touched again i think i read in LKML.) But it is good that the world is getting more perfect. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From roberto.foglietta at gmail.com Thu Sep 21 07:24:22 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Thu, 21 Sep 2023 09:24:22 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <20230921003426.-jVMT%steffen@sdaoden.eu> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> Message-ID: On Thu, 21 Sept 2023 at 02:35, Steffen Nurpmeso wrote: > > Steffen Nurpmeso wrote in > <20230919222910.pn44Y%steffen at sdaoden.eu>: > |Laurent Bercot wrote in > | : > ||>|IIRC writing to /dev/urandom doesn't do what you want it to do. > ||>|You have to use an ioctl() to actually set entropy. > ||> > ||>And that is the sad point about it. > ||>Kernel hackers should stand up to allow it again! > || > || As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that > ||any user can write to /dev/urandom, including malicious users, so > ||you cannot credit what they write. > | > |Well i was talking on unlocking the thing upon boot. Only root > |there is. > | > |And then you could easily look who is the writer, or require > |CAP_SYS_ADMIN or whatever when deciding whether "entropy" is to be > |counted or not. That is just a check that is done a thousand > |times in the kernel, i would think. > | > |Actually, if i recall correctly, it is only for unlocking after > |the rewrite anyway since entropy is not counted no more? At least > |the IOCTL which gives the number does not change no more. > | > |The thing is plain. Unix "everything is a file" ("except when it > |isn't"; except on Plan9, where it is) is broken by that for no > |reason. I did not read the files it is too late, but i would have > |stood up and have done it like that. > ... > ||[1]: https://lwn.net/ml/linux-kernel/YjqVemCkZCU1pOzj at mit.edu/ > ||[2]: https://lwn.net/ml/linux-kernel/YjqbcQbYHCOpgqGg at zx2c4.com/ > > Now i have read them and [2] indeed brings SYS_ADMIN into play but > disregards it with > > The problem with that approach, though, is that various > userspaces might already write garbage into /dev/urandom, not > expecting them to be credited -- for example, some userspace > hardware configuration component that writes some serial number > there. So I'm sort of hesitant to _change_ the behavior that > we've had for so long. > > and > > Another variation on that would be to do what this current patch > does, but only crng_pre_init_inject() on CAP_SYS_ADMIN. But this > has the same pitfall of only working as intended at cnrg_init=0 > but not crng_init=1. > > One could have made a survey? Before breaking all non-systemd > distributions i mean. > Change creditable root writes to /dev/random instead? I consider this interface one of the brokest seen in the UNIX domain ever. Despite this, I am reluctant in wishing that it would be completely changed because breaking the back-compatibility with 30years old production systems is something that goes beyond a survey. Everything can be improved and enhanced, obviously and I hope this will happen for this interface as well, in a smooth and back-compatible way. Fortunately, in the years the /sys became a standard and this allows a creation of a specific device u/random that has a property like entropy. That property presented like a file, despite the ownership and rwx bitmask assigned, it will check for CAP_SYSADMIN and allow root to inject directly in the entropy pool whatever data s/he wants and/or resetting the pool. This enanchement is good not only to re-established the principle for which everything in UNIX is a file and root is the Admin but also allows people to experiment with application running on a system for which the entropy pool has been arbitrarily changed including reseted to a large bounce of zeros. A condition that should never happen in production but could be very interesting to see in a lab experiment. After all, it would not be the first time that we design a way to inject errors into a system to debug / verify the error handlers or watch how the system and application behave in extreme situations or corner-cases. Nothing new under the sun. Right? IMHO, I vote for /sys rather than /proc/debug. The capability to directly handle the system entropy pool should be an Admin privilege even before being a debug option. As well as disable that handling with an echo 1 > /sys/driver/random/entropy_handle_disable In such a way that even Admin cannot do something like echo 1 > /sys/driver/random/entropy_pool_reset or dd if=/dev/zero count=128 of=/sys/driver/random/entropy_pool without enabling the pool handler interface which would come disabled by default. I also suggest that > (write) will reset it, while >> (append) will just append. The /sys interface for u/random is completely new and we can accept to establish a completely new standard about it. Following the common rules of dealing with files, it seems to me the most reasonable way to do it. I hope this helps, R- From yetanothergeek at gmail.com Thu Sep 21 12:10:49 2023 From: yetanothergeek at gmail.com (Jeff Pohlmeyer) Date: Thu, 21 Sep 2023 07:10:49 -0500 Subject: syslogd problem In-Reply-To: <2b869ce9-af91-81ab-038d-ba89ee7b334e@tls.msk.ru> References: <2b869ce9-af91-81ab-038d-ba89ee7b334e@tls.msk.ru> Message-ID: On Wed, Sep 20, 2023 at 11:54?AM Michael Tokarev wrote: > It's been broken for quite some time Thanks, I didn't know about that one. These are actually two different problems, your "read() not blocking" fix, and my "OPT_locallog not in opts" fix. After applying both fixes I was finally able to get syslogd working from my init script. Sure hope these can make it into master soon! - Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: syslogd-option_mask32.patch Type: text/x-patch Size: 468 bytes Desc: not available URL: From philipp_subx at redfish-solutions.com Thu Sep 21 15:06:29 2023 From: philipp_subx at redfish-solutions.com (Philip Prindeville) Date: Thu, 21 Sep 2023 09:06:29 -0600 Subject: Support -E in awk Message-ID: Hi, We have some scripts written in awk that we'd like to have take flags, but when the script is embedded in a hashbang file (#!/usr/bin/awk -f), then awk ends up consuming any arguments that look like flags, so we're limited in the compatibility we can provide. Looking at editors/awk.c there's a comment about adding -E being "easy-ish". Glad to hear it. We have an urgent need for this, and we're hoping it can be in the next release. We've opened a bug: https://bugs.busybox.net/show_bug.cgi?id=15781 How realistic is it to get this out for the next cycle? Thanks, -Philip From steffen at sdaoden.eu Thu Sep 21 18:05:26 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Thu, 21 Sep 2023 20:05:26 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> Message-ID: <20230921180526.3TPIY%steffen@sdaoden.eu> Roberto A. Foglietta wrote in : |On Thu, 21 Sept 2023 at 02:35, Steffen Nurpmeso wrote: |> |> Steffen Nurpmeso wrote in |> <20230919222910.pn44Y%steffen at sdaoden.eu>: |>|Laurent Bercot wrote in |>| : |>||>|IIRC writing to /dev/urandom doesn't do what you want it to do. |>||>|You have to use an ioctl() to actually set entropy. |>||> |>||>And that is the sad point about it. |>||>Kernel hackers should stand up to allow it again! |>|| |>|| As Ted Ts'o said[1], and Donenfeld agreed[2], the problem is that |>||any user can write to /dev/urandom, including malicious users, so |>||you cannot credit what they write. |>| |>|Well i was talking on unlocking the thing upon boot. Only root |>|there is. |>| |>|And then you could easily look who is the writer, or require ... |>|The thing is plain. Unix "everything is a file" ("except when it |>|isn't"; except on Plan9, where it is) is broken by that for no |>|reason. I did not read the files it is too late, but i would have |>|stood up and have done it like that. |> ... |>||[1]: https://lwn.net/ml/linux-kernel/YjqVemCkZCU1pOzj at mit.edu/ |>||[2]: https://lwn.net/ml/linux-kernel/YjqbcQbYHCOpgqGg at zx2c4.com/ |> |> Now i have read them and [2] indeed brings SYS_ADMIN into play but |> disregards it with |> |> The problem with that approach, though, is that various |> userspaces might already write garbage into /dev/urandom, not |> expecting them to be credited -- for example, some userspace |> hardware configuration component that writes some serial number |> there. So I'm sort of hesitant to _change_ the behavior that |> we've had for so long. ... |> One could have made a survey? Before breaking all non-systemd |> distributions i mean. |> Change creditable root writes to /dev/random instead? | |I consider this interface one of the brokest seen in the UNIX domain I want to add that [2] first said Since we now always use a cryptographic hash function, we can haphazardly mix whatever any user wants, without too much concern. The issue is whether we _credit_ those bits. Were we to credit those bits, a malicious unpriv'd user could as well as that you removed the quote Before i had to install haveged because python2 via mailman consumes so much random you get grazy. and remark that haveged creates many (many!!) bits of entropy in the fraction of a second. Luckily i could remove haveged after the Linux random rewrite. The problem is (thus) solely, in an asynchronous startup, when the CRNG reports itself seeded (and security related software that hang waiting on it for this get served). ... |IMHO, I vote for /sys rather than /proc/debug. The capability to There is /proc/sys/kernel/random. |directly handle the system entropy pool should be an Admin privilege |even before being a debug option. As well as disable that handling |with an | |echo 1 > /sys/driver/random/entropy_handle_disable | |In such a way that even Admin cannot do something like | |echo 1 > /sys/driver/random/entropy_pool_reset | |or | |dd if=/dev/zero count=128 of=/sys/driver/random/entropy_pool ... Access to the pool will never happen again i bet. Personally i think it is best effort and that this was achieved. I was convinced of the rewrite of Donenfeld, including the forward secrecy that Bernstein was talking about -- i cannot reference, anything under cr.yp.to/crypto.html seem to have disappeared. I found this large pool mixing that was used before and is still used by OpenBSD a bit suspect. On the other hand a larger pool can store more input. Hm. (I find it grazy how expensively this is handled even. It happens all the time while i write this, .. WOW! ...) One thing is plain, i would not know how to make it any better. Except for the topic of this thread. And i have to concur that with the seedrng approach super fine- grained and secure locking in a super-parallel super-fast async boot process is possible. But i do not get his arguments, who writes during the boot, and so much, and if, just change to /dev/random for credit (i mean, isn't /dev/urandom the not-so-good thing anyhow). But if it is not that (random), then a trigger like yours as in /proc/sys/kernel/random/pool_is_seeded is not a bad idea. Then a boot sequence could decide when to unlock daemons that yet accumulated. This is of course all hypothetical. And in the meantime most boot sequences adopted a dedicated program to be shipped. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From roberto.foglietta at gmail.com Sat Sep 23 05:51:04 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sat, 23 Sep 2023 07:51:04 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <20230921180526.3TPIY%steffen@sdaoden.eu> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> <20230921180526.3TPIY%steffen@sdaoden.eu> Message-ID: On Thu, 21 Sept 2023 at 20:05, Steffen Nurpmeso wrote: > |IMHO, I vote for /sys rather than /proc/debug. The capability to > > There is /proc/sys/kernel/random. > > |directly handle the system entropy pool should be an Admin privilege > |even before being a debug option. As well as disable that handling > |with an > | > |echo 1 > /sys/driver/random/entropy_handle_disable > | > |In such a way that even Admin cannot do something like > | > |echo 1 > /sys/driver/random/entropy_pool_reset > | > |or > | > |dd if=/dev/zero count=128 of=/sys/driver/random/entropy_pool > ... > > Access to the pool will never happen again i bet. Well, not by /dev/random or by /dev/urandom for a matter of back-compatibility and I agree on this. Breaking well established interfaces is not a good policy especially when an alternative is possible. > And i have to concur that with the seedrng approach super fine- > grained and secure locking in a super-parallel super-fast async > boot process is possible. Mixing parallel output streams into a single one like a log daemon does, for a 4 or a 8 cores system is a great source of entropy. On the other side, you can argue that each line of those streams' line carries on a timestamp and this strictly enforces an order. As long as the timestamp is fine-grained, it supplies greatly the lack of arbitrary mixing where arbitrary means when the scheduler decides to switch to another task. > But i do not get his arguments, who writes during the boot, and so > much, and if, just change to /dev/random for credit (i mean, isn't > /dev/urandom the not-so-good thing anyhow). > But if it is not that (random), then a trigger like yours as in > /proc/sys/kernel/random/pool_is_seeded is not a bad idea. Thanks Steffen. After all, it is nothing more that a generalisation of an interface to inject errors which usually would stay in debugfs but because can also have a real-world production advantage - let the admin dealing with the entropy in a straightforward manner - it could be a great way to overcome the limitations of the ioctl() interface, in some advanced system (most of the system can be considered advanced by this definition). Recently I saw a commercial about a mother board carrying up to 8 SoC each of one with a 4-cores ARM CPU, for a total of 32 cores. That board fully equipped was selling for less than $1000. High parallel computing entrance barrier is not anymore as expensive as in the past, but quite cheap. At least for everyone that can pay for a top-gamma smartphone ($1000-$2000) and with the same money can enter into HP computing. Best regards, R- From nero at w1r3.net Sat Sep 23 11:50:04 2023 From: nero at w1r3.net (Nero) Date: Sat, 23 Sep 2023 11:50:04 +0000 Subject: [PATCH] install: Fix chown resetting suid/sgid bits from chmod Message-ID: Since Linux 2.2.13, chown(2) resets the suid/gid bits for all users. This patch changes the ordering so that chmod gets called after chown. This behavior follows GNU coreutils. Signed-off-by: Nero --- coreutils/install.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/coreutils/install.c b/coreutils/install.c index c0f1c538a..a81a5a1ef 100644 --- a/coreutils/install.c +++ b/coreutils/install.c @@ -244,6 +244,14 @@ int install_main(int argc, char **argv) } } + /* Set the user and group id */ + if ((opts & (OPT_OWNER|OPT_GROUP)) + && lchown(dest, uid, gid) == -1 + ) { + bb_perror_msg("can't change %s of %s", "ownership", dest); + ret = EXIT_FAILURE; + } + /* Set the file mode (always, not only with -m). * GNU coreutils 6.10 is not affected by umask. */ if (chmod(dest, mode) == -1) { @@ -254,13 +262,6 @@ int install_main(int argc, char **argv) if (use_default_selinux_context) setdefaultfilecon(dest); #endif - /* Set the user and group id */ - if ((opts & (OPT_OWNER|OPT_GROUP)) - && lchown(dest, uid, gid) == -1 - ) { - bb_perror_msg("can't change %s of %s", "ownership", dest); - ret = EXIT_FAILURE; - } next: if (ENABLE_FEATURE_CLEAN_UP && isdir) free(dest); -- 2.42.0 From steffen at sdaoden.eu Sat Sep 23 14:01:26 2023 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Sat, 23 Sep 2023 16:01:26 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> <20230921180526.3TPIY%steffen@sdaoden.eu> Message-ID: <20230923140126.8Rn85%steffen@sdaoden.eu> Roberto A. Foglietta wrote in : |On Thu, 21 Sept 2023 at 20:05, Steffen Nurpmeso wrote: |>|IMHO, I vote for /sys rather than /proc/debug. The capability to There already _is_ a debugfs interface? This must be new then. I do not have debugfs no more, it is of no use for me. Sorry if i have missed anything. ... |>|directly handle the system entropy pool should be an Admin privilege ... |> Access to the pool will never happen again i bet. | |Well, not by /dev/random or by /dev/urandom for a matter of |back-compatibility and I agree on this. Breaking well established |interfaces is not a good policy especially when an alternative is |possible. There was a pool access ioctl that was removed. Must i comment? I have no problem with that, even though one could say "i have a very good entropy generator" and wants to stir the pool her/himself. Direct I/O and even mapping of physical pages in userspace is possible / discussed, so why that not? Concurrent unlocked messing via binary ops surely improves the randomness (:-} ... |but quite cheap. At least for everyone that can pay for a top-gamma |smartphone ($1000-$2000) and with the same money can enter into HP |computing. Even the newest fairphone is only 70% fair iirc. No. Like the wonderful Zarah Leander sang in "Nur nicht aus Liebe weinen" (badly: [anything, but] not crying due to love): the answer is always renunciation. (That was, granted, under the Nazi regime.) (And, it counteracts a bit with Hans Albers, "Hoppla! Jetzt komm' ich", maybe.) But now clear the streets, for me! --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From roberto.foglietta at gmail.com Sat Sep 23 14:54:32 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sat, 23 Sep 2023 16:54:32 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: <20230923140126.8Rn85%steffen@sdaoden.eu> References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> <20230921180526.3TPIY%steffen@sdaoden.eu> <20230923140126.8Rn85%steffen@sdaoden.eu> Message-ID: On Sat, 23 Sept 2023 at 16:30, Steffen Nurpmeso wrote: > > Roberto A. Foglietta wrote in > : > |On Thu, 21 Sept 2023 at 20:05, Steffen Nurpmeso wrote: > |>|IMHO, I vote for /sys rather than /proc/debug. The capability to > > There already _is_ a debugfs interface? Nothing which is interesting, AFAIK redfishos:~ # find /sys -name rand\* /sys/class/mem/random /sys/devices/virtual/mem/random /sys/module/random redfishos:~ # cat /sys/class/mem/random/dev 1:8 redfishos:~ # cat /sys/devices/virtual/mem/random/uevent MAJOR=1 MINOR=8 DEVNAME=random DEVMODE=0666 redfishos:~ # cat /sys/module/random/parameters/ratelimit_disable 0 Until someone add a patch that creates something like entropy_pool_access_status (RW) cat: disable write append hashing (were hashing is like append but hashing before) entropy_pool_access (RW) cat: number of bytes of data, number of entropy bit > This must be new then. Just a couple more of parameters > I do not have debugfs no more, it is of no > use for me. Sorry if i have missed anything. You are welcome, do not worry about. Best regards, R- From roberto.foglietta at gmail.com Sat Sep 23 22:30:03 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Sun, 24 Sep 2023 00:30:03 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> <20230921180526.3TPIY%steffen@sdaoden.eu> <20230923140126.8Rn85%steffen@sdaoden.eu> Message-ID: On Sat, 23 Sept 2023 at 16:54, Roberto A. Foglietta wrote: > > On Sat, 23 Sept 2023 at 16:30, Steffen Nurpmeso wrote: > > > > Roberto A. Foglietta wrote in > > : > > |On Thu, 21 Sept 2023 at 20:05, Steffen Nurpmeso wrote: > > |>|IMHO, I vote for /sys rather than /proc/debug. The capability to > > > > There already _is_ a debugfs interface? > > Nothing which is interesting, AFAIK > However, I think that on the new hardware the entropy creation is not anymore an issue https://www.kernel.org/doc/html/latest/admin-guide/hw_random.html In both my laptop and smartphone on which I develop, the /dev/hwrng exists. Well, not totally solve the problem: http://main.lv/writeup/kernel_dev_hwrng.md <> Best regards, R- From roberto.foglietta at gmail.com Mon Sep 25 03:34:58 2023 From: roberto.foglietta at gmail.com (Roberto A. Foglietta) Date: Mon, 25 Sep 2023 05:34:58 +0200 Subject: RFC: initialize /dev/urandom, is it necessary? Can we do it in a better way? In-Reply-To: References: <2913c3ed-3918-4b74-5db8-dc1cb64da5e2@intellitree.com> <4b5be2ae-ff25-30b2-7fdd-5379f4d81680@intellitree.com> <1628d46df2fb4580b474fd0ea43a5778@AcuMS.aculab.com> <20230919185453.WtzGc%steffen@sdaoden.eu> <20230919222910.pn44Y%steffen@sdaoden.eu> <20230921003426.-jVMT%steffen@sdaoden.eu> Message-ID: On Thu, 21 Sept 2023 at 09:24, Roberto A. Foglietta wrote: > > I consider this interface one of the brokest seen in the UNIX domain > ever. Despite this, I am reluctant in wishing that it would be > completely changed because breaking the back-compatibility with > 30years old production systems is something that goes beyond a survey. > Everything can be improved and enhanced, obviously and I hope this > will happen for this interface as well, in a smooth and > back-compatible way. > This, as long as we are writing about computer information technology, Linux in this case. If you ask me about society, my opinion is completely different. The reason is pretty simple: computers are "machines" and until the day they will develop an AGI and show to be a sentient self-conscious entity?es, they have no rights but just work to do - no matter how this work is broken or a burden. Put your chips on fire! When we talk about society, we are speaking about people, about human rights, about giving a meaning to our development and progress (aka society). The two concepts "technology" and "people" are not even comparable. I wrote this because I had the sensation that someone confused the two aspects. Some people never change their mind, whatever. While we can always reinstall an old machine to update it. The two concepts "people" and "technology" are way different in good and bad shapes. KISS works for IT, but rarely it works for people. etc. etc. - Just to clarify and to avoid that someone can leverage my writing to extend their meaning out of their specific context to influence other people's opinion or trying to understand my "under the hood" thinking. There is no doubt that I always was, I am and I will always be in favour of progress (a change for the better, a solid true improvement based on facts and not just opinions) when we are talking about people and society, no matter what, no matter who. From online at chave.us Tue Sep 26 23:38:56 2023 From: online at chave.us (Carl Chave) Date: Tue, 26 Sep 2023 19:38:56 -0400 Subject: Question on mount helpers Message-ID: On Alpine linux, busybox is configured with CONFIG_FEATURE_MOUNT_HELPERS=y and a nilfs2 filesystem does successfully mount from fstab with busybox mount, however, the nilfs_cleanerd process is not automatically started. From the nilfs_cleanerd documentation: > This program can be invoked either automatically by mount.nilfs2(8) or manually by an administrator. However, users are recommended to invoke nilfs_cleanerd through mount.nilfs2(8) or mount(8) and shutdown it through umount.nilfs2(8) or umount(8) in order to avoid state inconsistencies among administration tools. Switching to util-linux mount does result in nilfs_cleanerd being started automatically so my tentative conclusion is that util-linux mount will always use a mount helper when available (unless called with -i) and busybox mount will use the helper only after a "generic" mount attempt fails. Is that correct? If so, would you consider reversing the order? Thanks, Carl From tdtemccnp at gmail.com Fri Sep 29 14:15:07 2023 From: tdtemccnp at gmail.com (Turritopsis Dohrnii Teo En Ming) Date: Fri, 29 Sep 2023 22:15:07 +0800 Subject: Apache web server httpd in Netgear Wireless Router WNR3500L Message-ID: Subject: Apache web server httpd in Netgear Wireless Router WNR3500L Good day from Singapore, Release version : Netgear Wireless Router WNR3500L U12H13699/V1.0.2.35/32.1.9_TR069_0528 Time : May 29 2010 10:31:41 CFE version : v1.0.36 We were trying to login to the Netgear wireless router but the web UI was not available. Then we managed to telnet (TCP port 23) into the Netgear wireless router. The operating system is BusyBox. I am sorry I did not note down the version of BusyBox. BusyBox is a stripped-down version of the Linux operating system. When I run the Linux command "httpd", the Apache web server started and we managed to login to the web UI already. But when we need to reboot the Netgear wireless router, where are the Apache web server httpd autostart scripts? Do we have to manually execute the httpd command every time we reboot the Netgear wireless router? Why was httpd not running when we tried to login to the web UI? Did somebody kill the httpd process or deleted the httpd autostart scripts? Please advise. Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore Blogs: https://tdtemcerts.blogspot.com https://tdtemcerts.wordpress.com GIMP also stands for Government-Induced Medical Problems. I have configured, setup and deployed 22 Fortigate firewall appliances for 20 companies/organizations in Singapore so far as of 29 Sep 2023 Fri. From yetanothergeek at gmail.com Fri Sep 29 14:44:47 2023 From: yetanothergeek at gmail.com (Jeff Pohlmeyer) Date: Fri, 29 Sep 2023 09:44:47 -0500 Subject: Apache web server httpd in Netgear Wireless Router WNR3500L In-Reply-To: References: Message-ID: On Fri, Sep 29, 2023 at 9:16?AM Turritopsis Dohrnii Teo En Ming wrote: > Subject: Apache web server httpd in Netgear Wireless Router WNR3500L > where are the Apache web server httpd autostart scripts? > Why was httpd not running when we tried to login to the web UI? Busybox does not provide any "autostart scripts" or "web UI". You might do better to ask here: https://community.netgear.com/ - Jeff From tdtemccnp at gmail.com Fri Sep 29 14:51:38 2023 From: tdtemccnp at gmail.com (Turritopsis Dohrnii Teo En Ming) Date: Fri, 29 Sep 2023 22:51:38 +0800 Subject: Apache web server httpd in Netgear Wireless Router WNR3500L In-Reply-To: References: Message-ID: On Fri, 29 Sept 2023 at 22:44, Jeff Pohlmeyer wrote: > > On Fri, Sep 29, 2023 at 9:16?AM Turritopsis Dohrnii Teo En Ming > wrote: > > > Subject: Apache web server httpd in Netgear Wireless Router WNR3500L > > where are the Apache web server httpd autostart scripts? > > Why was httpd not running when we tried to login to the web UI? > > Busybox does not provide any "autostart scripts" or "web UI". You > might do better to ask here: https://community.netgear.com/ > > - Jeff Noted. I shall do so. Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore