visitor (0 QPoints)
  • FR
  • EN
  • NL
  • DE
  • ES
315 experts, 1193 registered users, 1659 questions already answered
European Experts Exchange, the very best site for high-quality IT solutions

New Improved Search!

 


05/10/2011 1h30 : Steve Jobs is dead, the father of Apple ][ is gone, we are all orphaned.

Languages :: PHP :: Source Code of specified URL into a variable
By: blacksix U.S.A.  Date: 15/04/2003 00:00:00  English  Points: 400 Status: Answered
Quality : Excellent
I'm looking to grab the source code from a website and put it into a variable so I can then go through and pick up a certain chunk of data and format it to fit my site format.... For example say i want to grab the top 10 web searches from a weekly report on lycose.com... i say something like
$top10html = GETURL("<A HREF="http://lycose.com/blablabla/top10/index.html">http://lycose.com/blablabla/top10/index.html</a>");
and then work with the $top10html variable from there....

I've got no idea where to start...
I've heard of CURL and fsockopen (i think) but dont really get them.. I just need a simple explaination lol

Thanks!!
By: cirtap Date: 15/04/2003 11:04:00 English  Type : Comment
Hi blacksix,
do you think it's a good idea to "steal" someone else's content without permission?

CirTap
By: blacksix Date: 15/04/2003 12:02:00 English  Type : Comment
No I do not think that is a good or moral idea. I'm not doing this do "Steal" someone elses work... I design websites for an Irish record company and would like to post the schedule information from websites of the artist websites on our record labels. We have permission. We want to make one centralized website with the info, and the artists think its a good idea as well. The lycose thing was an example, and not my intention whatsoever. Sure I could like their sites, but my boss said he wants to keep it all in the same format, and what he asks for, it is my responsibility to produce. There may be other ways to do the same thing involving a centralized database of some sort but thats just too hard to set up across the different sites. Sorry, I should have clarified this.
By: blacksix Date: 15/04/2003 15:51:00 English  Type : Comment
NEVERMIND I figured out how to do it using fopen() and fread()
By: lupuleine Date: 16/04/2003 05:23:00 English  Type : Comment
hi

so here's what you could do

$lines = @file("<A HREF="http://lycose.com/blablabla/top10/index.html">http://lycose.com/blablabla/top10/index.html</a>");
then with an loop like

for($i=0;$i<sizeof($lines);$i++)
{
//in $lines[$i] you have every single source code line
}
or you can use while loop
and you check each line for the information you want
it's ok to look for special tags that are around the information you look for

Ps.
Cirtap if really don't wanna help people around this forum, then please don't post any crap. It would be appreciated. 10x
By: VGR Date: 16/04/2003 05:31:00 English  Type : Comment
agree with both, in fact 8-)

Even if I'm a "stealer" ;-)

It's not the source code of the site that gets retrieved (and not "stolen") but the HTML generated by it ; this is publicly displayed, after all, and any browser can read it, so why not a PHP script ?

I think you figured it out ; fine. All you've left is to parse the results for the data you search. I may help there too, althought thisis a very common question whose answer is in the recent PAQ ;-)
By: TheFalklands Date: 16/04/2003 19:04:00 English  Type : Comment
function GetUrl($url)
{
$lines = file($url);
for($i=0;$i<sizeof($lines);$i++)
{
$all_lines .= $lines[$i];
}
return $all_lines;
}

$url = "<A HREF="http://lycose.com/blablabla/top10/index.html">http://lycose.com/blablabla/top10/index.html</a>";
$top10html = GetUrl($url);
By: VGR Date: 16/04/2003 19:14:00 English  Type : Comment
yes, yes, yes everybody knows how to do that :-)

the difficult part is to parse efficiently (and clearly, for you regexp guys around there :D ) the $all_lines[]
By: bljak Date: 16/04/2003 21:57:00 English  Type : Comment
tsk tsk
not good idea saying CirTap doesn't help, if you have had took a look at his profile before saying that you would know better, however, has nothing to do with this question, i just gone mad

p.s. god bless regexps :P

//bljak
By: lupuleine Date: 16/04/2003 22:38:00 English  Type : Comment
I looked at cirtap's profile and I respect his/her knowledge but, the given answer was not for help to anyone on this forum. Sorry, I only say what I have to say. ;)
By: blacksix Date: 16/04/2003 23:19:00 English  Type : Comment
so I got it working before i got an answer and i cant delete the question so im not just gonna toss those points... althought I appreciate the effort, I figure i'll ask another question here if its not agains some kind of rule...

This is a very slow script.. so what i would like to do is write a script that grabs the html chunk every couple hours (without being manually accessed) and stores it locally for much faster retrieval. How would I start this? I know this kind of thing can be done with databases so I figure it can be done here too... THANKS GUYS!!
By: VGR Date: 16/04/2003 23:25:00 English  Type : Comment
easy. You can program the retrieval to take place every 2 hours using my free RobotOuaibe utility, or (if PHP is cgi exec) using AT or the crontab, depending on your platform.

You can also (solution of the poor :D ) leave open a browser window with a refresh time of 7200 seconds ;-)

interested ?
By: VGR Date: 16/04/2003 23:26:00 English  Type : Comment
as for your question, can't you reduce thepoints yourself ? Or ask for a refund in feedback area ?
By: blacksix Date: 16/04/2003 23:45:00 English  Type : Comment
thanks VGR
its UNIX so thats crontab right?
can you tell me a little about it? The server admin has to set that up correct? The points are yours just point me in the right direction. Thanks once again
By: bljak Date: 16/04/2003 23:54:00 English  Type : Comment
no need for your admin to set it up
you can do that yourself
man crontab
and then use it

//bljak
By: VGR Date: 17/04/2003 00:02:00 English  Type : Answer
yes
the crontab is usually manipulated, if my souvenirs are correct, with command :
crontab

(it konws which user you are and the entries you'lla dd will be yours only)

Note that this needsto be applied using commands like :
php nameofscript.php
By: blacksix Date: 17/04/2003 00:06:00 English  Type : Comment
VGR is da man
By: bljak Date: 17/04/2003 00:10:00 English  Type : Comment
is a man?
damn, i already tried to flirt with him, thinking its woman :P

//bljak
By: VGR Date: 17/04/2003 00:28:00 English  Type : Comment
huh huh huh
where do you come from bljak?

Do register to be able to answer

©2010 These pages are served without commercial sponsorship. (No popup ads, etc...). Bandwidth abuse increases hosting cost forcing sponsorship or shutdown. This server aggressively defends against automated copying for any reason including offline viewing, duplication, etc... Please respect this requirement and DO NOT RIP THIS SITE.
Please DO link to this page! Post to Twitter Post to Plurk Post to Yahoo Buzz Post to Delicious Post to Digg Post to Facebook Post to MySpace Post to Ping.fm Post to Reddit Post to StumbleUpon

EContact
browser fav
page generated in 85.091830 milliseconds

Why Google AdSense ads ?
compteur
 Ranking-Hits PageRank for this page